In this notebook, I'll lead you through using TensorFlow to implement the word2vec algorithm using the skip-gram architecture. By implementing this, you'll learn about embedding words for use in natural language processing. This will come in handy when dealing with things like translations.
Here are the resources I used to build this notebook. I suggest reading these either beforehand or while you're working on this material.
When you're dealing with language and words, you end up with tens of thousands of classes to predict, one for each word. Trying to one-hot encode these words is massively inefficient, you'll have one element set to 1 and the other 50,000 set to 0. The word2vec algorithm finds much more efficient representations by finding vectors that represent the words. These vectors also contain semantic information about the words. Words that show up in similar contexts, such as "black", "white", and "red" will have vectors near each other. There are two architectures for implementing word2vec, CBOW (Continuous Bag-Of-Words) and Skip-gram.
In this implementation, we'll be using the skip-gram architecture because it performs better than CBOW. Here, we pass in a word and try to predict the words surrounding it in the text. In this way, we can train the network to learn representations for words that show up in similar contexts.
First up, importing packages.
In [1]:
import time
import numpy as np
import tensorflow as tf
import utils
Load the text8 dataset, a file of cleaned up Wikipedia articles from Matt Mahoney. The next cell will download the data set to the data folder. Then you can extract it and delete the archive file to save storage space.
In [2]:
from urllib.request import urlretrieve
from os.path import isfile, isdir
from tqdm import tqdm
import zipfile
dataset_folder_path = 'data'
dataset_filename = 'text8.zip'
dataset_name = 'Text8 Dataset'
class DLProgress(tqdm):
last_block = 0
def hook(self, block_num=1, block_size=1, total_size=None):
self.total = total_size
self.update((block_num - self.last_block) * block_size)
self.last_block = block_num
if not isfile(dataset_filename):
with DLProgress(unit='B', unit_scale=True, miniters=1, desc=dataset_name) as pbar:
urlretrieve(
'http://mattmahoney.net/dc/text8.zip',
dataset_filename,
pbar.hook)
if not isdir(dataset_folder_path):
with zipfile.ZipFile(dataset_filename) as zip_ref:
zip_ref.extractall(dataset_folder_path)
with open('data/text8') as f:
text = f.read()
Text8 Dataset: 31.4MB [00:14, 2.11MB/s]
Here I'm fixing up the text to make training easier. This comes from the utils module I wrote. The preprocess function coverts any punctuation into tokens, so a period is changed to <PERIOD>. In this data set, there aren't any periods, but it will help in other NLP problems. I'm also removing all words that show up five or fewer times in the dataset. This will greatly reduce issues due to noise in the data and improve the quality of the vector representations. If you want to write your own functions for this stuff, go for it.
In [3]:
words = utils.preprocess(text)
print(words[:30])
['anarchism', 'originated', 'as', 'a', 'term', 'of', 'abuse', 'first', 'used', 'against', 'early', 'working', 'class', 'radicals', 'including', 'the', 'diggers', 'of', 'the', 'english', 'revolution', 'and', 'the', 'sans', 'culottes', 'of', 'the', 'french', 'revolution', 'whilst']
In [4]:
print(words[-30:])
['today', 'platform', 'access', 'to', 'the', 'fortress', 'the', 'site', 'of', 'masada', 'was', 'identified', 'in', 'one', 'eight', 'four', 'two', 'and', 'extensively', 'excavated', 'in', 'one', 'nine', 'six', 'three', 'one', 'nine', 'six', 'five', 'b']
In [5]:
print("Total words: {}".format(len(words)))
print("Unique words: {}".format(len(set(words))))
Total words: 16680599
Unique words: 63641
And here I'm creating dictionaries to covert words to integers and backwards, integers to words. The integers are assigned in descending frequency order, so the most frequent word ("the") is given the integer 0 and the next most frequent is 1 and so on. The words are converted to integers and stored in the list int_words.
In [6]:
vocab_to_int, int_to_vocab = utils.create_lookup_tables(words)
int_words = [vocab_to_int[word] for word in words]
In [7]:
int_to_vocab
Out[7]:
{0: 'the',
1: 'of',
2: 'and',
3: 'one',
4: 'in',
5: 'a',
6: 'to',
7: 'zero',
8: 'nine',
9: 'two',
10: 'is',
11: 'as',
12: 'eight',
13: 'for',
14: 's',
15: 'five',
16: 'three',
17: 'was',
18: 'by',
19: 'that',
20: 'four',
21: 'six',
22: 'seven',
23: 'with',
24: 'on',
25: 'are',
26: 'it',
27: 'from',
28: 'or',
29: 'his',
30: 'an',
31: 'be',
32: 'this',
33: 'which',
34: 'at',
35: 'he',
36: 'also',
37: 'not',
38: 'have',
39: 'were',
40: 'has',
41: 'but',
42: 'other',
43: 'their',
44: 'its',
45: 'first',
46: 'they',
47: 'some',
48: 'had',
49: 'all',
50: 'more',
51: 'most',
52: 'can',
53: 'been',
54: 'such',
55: 'many',
56: 'who',
57: 'new',
58: 'used',
59: 'there',
60: 'after',
61: 'when',
62: 'into',
63: 'american',
64: 'time',
65: 'these',
66: 'only',
67: 'see',
68: 'may',
69: 'than',
70: 'world',
71: 'i',
72: 'b',
73: 'would',
74: 'd',
75: 'no',
76: 'however',
77: 'between',
78: 'about',
79: 'over',
80: 'years',
81: 'states',
82: 'people',
83: 'war',
84: 'during',
85: 'united',
86: 'known',
87: 'if',
88: 'called',
89: 'use',
90: 'th',
91: 'system',
92: 'often',
93: 'state',
94: 'so',
95: 'history',
96: 'will',
97: 'up',
98: 'while',
99: 'where',
100: 'city',
101: 'being',
102: 'english',
103: 'then',
104: 'any',
105: 'both',
106: 'under',
107: 'out',
108: 'made',
109: 'well',
110: 'her',
111: 'e',
112: 'number',
113: 'government',
114: 'them',
115: 'm',
116: 'later',
117: 'since',
118: 'him',
119: 'part',
120: 'name',
121: 'c',
122: 'century',
123: 'through',
124: 'because',
125: 'x',
126: 'university',
127: 'early',
128: 'life',
129: 'british',
130: 'year',
131: 'like',
132: 'same',
133: 'including',
134: 'became',
135: 'example',
136: 'day',
137: 'each',
138: 'even',
139: 'work',
140: 'language',
141: 'although',
142: 'several',
143: 'form',
144: 'john',
145: 'u',
146: 'national',
147: 'very',
148: 'much',
149: 'g',
150: 'french',
151: 'before',
152: 'general',
153: 'what',
154: 't',
155: 'against',
156: 'n',
157: 'high',
158: 'links',
159: 'could',
160: 'based',
161: 'those',
162: 'now',
163: 'second',
164: 'de',
165: 'music',
166: 'another',
167: 'large',
168: 'she',
169: 'f',
170: 'external',
171: 'german',
172: 'different',
173: 'modern',
174: 'great',
175: 'do',
176: 'common',
177: 'set',
178: 'list',
179: 'south',
180: 'series',
181: 'major',
182: 'game',
183: 'power',
184: 'long',
185: 'country',
186: 'king',
187: 'law',
188: 'group',
189: 'film',
190: 'still',
191: 'until',
192: 'north',
193: 'international',
194: 'term',
195: 'we',
196: 'end',
197: 'book',
198: 'found',
199: 'own',
200: 'political',
201: 'party',
202: 'order',
203: 'usually',
204: 'president',
205: 'church',
206: 'you',
207: 'death',
208: 'theory',
209: 'area',
210: 'around',
211: 'include',
212: 'god',
213: 'ii',
214: 'way',
215: 'did',
216: 'military',
217: 'population',
218: 'using',
219: 'though',
220: 'small',
221: 'following',
222: 'within',
223: 'non',
224: 'human',
225: 'left',
226: 'main',
227: 'among',
228: 'point',
229: 'r',
230: 'due',
231: 'p',
232: 'considered',
233: 'public',
234: 'popular',
235: 'computer',
236: 'west',
237: 'family',
238: 'east',
239: 'information',
240: 'important',
241: 'european',
242: 'man',
243: 'sometimes',
244: 'right',
245: 'old',
246: 'free',
247: 'word',
248: 'without',
249: 'last',
250: 'us',
251: 'members',
252: 'given',
253: 'times',
254: 'roman',
255: 'make',
256: 'h',
257: 'age',
258: 'place',
259: 'l',
260: 'thus',
261: 'science',
262: 'case',
263: 'become',
264: 'systems',
265: 'union',
266: 'born',
267: 'york',
268: 'line',
269: 'countries',
270: 'does',
271: 'isbn',
272: 'st',
273: 'control',
274: 'various',
275: 'others',
276: 'house',
277: 'article',
278: 'island',
279: 'should',
280: 'led',
281: 'back',
282: 'period',
283: 'player',
284: 'europe',
285: 'languages',
286: 'central',
287: 'water',
288: 'few',
289: 'western',
290: 'home',
291: 'began',
292: 'generally',
293: 'less',
294: 'k',
295: 'similar',
296: 'written',
297: 'original',
298: 'best',
299: 'must',
300: 'according',
301: 'school',
302: 'france',
303: 'air',
304: 'single',
305: 'force',
306: 'v',
307: 'land',
308: 'groups',
309: 'down',
310: 'how',
311: 'works',
312: 'development',
313: 'official',
314: 'support',
315: 'england',
316: 'j',
317: 'rather',
318: 'space',
319: 'data',
320: 'greek',
321: 'km',
322: 'named',
323: 'germany',
324: 'just',
325: 'games',
326: 'said',
327: 'version',
328: 'late',
329: 'earth',
330: 'company',
331: 'every',
332: 'economic',
333: 'short',
334: 'published',
335: 'black',
336: 'army',
337: 'off',
338: 'london',
339: 'million',
340: 'body',
341: 'field',
342: 'christian',
343: 'either',
344: 'empire',
345: 'social',
346: 'o',
347: 'developed',
348: 'standard',
349: 'court',
350: 'service',
351: 'kingdom',
352: 'along',
353: 'college',
354: 'republic',
355: 'sea',
356: 'america',
357: 'today',
358: 'result',
359: 'held',
360: 'team',
361: 'light',
362: 'means',
363: 'never',
364: 'especially',
365: 'third',
366: 'further',
367: 'forces',
368: 'character',
369: 'take',
370: 'men',
371: 'society',
372: 'show',
373: 'open',
374: 'possible',
375: 'fact',
376: 'battle',
377: 'took',
378: 'former',
379: 'books',
380: 'soviet',
381: 'river',
382: 'children',
383: 'having',
384: 'good',
385: 'local',
386: 'current',
387: 'son',
388: 'process',
389: 'natural',
390: 'present',
391: 'himself',
392: 'islands',
393: 'total',
394: 'near',
395: 'white',
396: 'days',
397: 'person',
398: 'itself',
399: 'seen',
400: 'culture',
401: 'little',
402: 'above',
403: 'software',
404: 'largest',
405: 'words',
406: 'upon',
407: 'level',
408: 'father',
409: 'side',
410: 'created',
411: 'red',
412: 'references',
413: 'press',
414: 'full',
415: 'region',
416: 'almost',
417: 'al',
418: 'image',
419: 'famous',
420: 'play',
421: 'came',
422: 'role',
423: 'once',
424: 'certain',
425: 'league',
426: 'jewish',
427: 'james',
428: 'january',
429: 'site',
430: 'again',
431: 'art',
432: 'numbers',
433: 'member',
434: 'areas',
435: 'movement',
436: 'religious',
437: 'type',
438: 'march',
439: 'community',
440: 'story',
441: 'played',
442: 'production',
443: 'released',
444: 'center',
445: 'rights',
446: 'real',
447: 'related',
448: 'foreign',
449: 'low',
450: 'ancient',
451: 'terms',
452: 'view',
453: 'source',
454: 'act',
455: 'minister',
456: 'change',
457: 'energy',
458: 'produced',
459: 'research',
460: 'actor',
461: 'making',
462: 'civil',
463: 'december',
464: 'women',
465: 'special',
466: 'style',
467: 'william',
468: 'design',
469: 'japanese',
470: 'available',
471: 'chinese',
472: 'forms',
473: 'canada',
474: 'northern',
475: 'died',
476: 'class',
477: 'living',
478: 'next',
479: 'particular',
480: 'program',
481: 'council',
482: 'television',
483: 'head',
484: 'david',
485: 'china',
486: 'middle',
487: 'established',
488: 'bc',
489: 'hand',
490: 'far',
491: 'july',
492: 'function',
493: 'position',
494: 'y',
495: 'built',
496: 'george',
497: 'band',
498: 'together',
499: 'w',
500: 'latin',
501: 'thought',
502: 'eastern',
503: 'charles',
504: 'parts',
505: 'instead',
506: 'study',
507: 'india',
508: 'might',
509: 'code',
510: 'included',
511: 'meaning',
512: 'trade',
513: 'per',
514: 'june',
515: 'least',
516: 'half',
517: 'model',
518: 'economy',
519: 'prime',
520: 'traditional',
521: 'always',
522: 'capital',
523: 'range',
524: 'emperor',
525: 'november',
526: 'young',
527: 'anti',
528: 'final',
529: 'text',
530: 'players',
531: 'uk',
532: 'april',
533: 'run',
534: 'september',
535: 'radio',
536: 'addition',
537: 'live',
538: 'august',
539: 'note',
540: 'taken',
541: 'italian',
542: 'lost',
543: 'nature',
544: 'project',
545: 'technology',
546: 'spanish',
547: 'october',
548: 'rate',
549: 'recent',
550: 'won',
551: 'true',
552: 'value',
553: 'uses',
554: 'russian',
555: 'est',
556: 'wrote',
557: 'effect',
558: 'album',
559: 'southern',
560: 'africa',
561: 'whose',
562: 'top',
563: 'historical',
564: 'australia',
565: 'catholic',
566: 'particularly',
567: 'self',
568: 'structure',
569: 'record',
570: 'evidence',
571: 'themselves',
572: 'rule',
573: 'influence',
574: 'cases',
575: 'subject',
576: 'referred',
577: 'continued',
578: 'nations',
579: 'below',
580: 'rock',
581: 'japan',
582: 'com',
583: 'song',
584: 'throughout',
585: 'names',
586: 'female',
587: 'title',
588: 'our',
589: 'therefore',
590: 'office',
591: 'star',
592: 'paul',
593: 'too',
594: 'cities',
595: 'february',
596: 'independent',
597: 'author',
598: 'problem',
599: 'species',
600: 'education',
601: 'done',
602: 'philosophy',
603: 'come',
604: 'higher',
605: 'originally',
606: 'market',
607: 'town',
608: 'my',
609: 'season',
610: 'love',
611: 'strong',
612: 'israel',
613: 'writer',
614: 'irish',
615: 'films',
616: 'elements',
617: 'robert',
618: 'whether',
619: 'despite',
620: 'eventually',
621: 'here',
622: 'football',
623: 'action',
624: 'internet',
625: 'sound',
626: 'individual',
627: 'network',
628: 'described',
629: 'practice',
630: 'characters',
631: 're',
632: 'royal',
633: 'la',
634: 'events',
635: 'formed',
636: 'commonly',
637: 'base',
638: 'received',
639: 'problems',
640: 'african',
641: 'food',
642: 'jews',
643: 'able',
644: 'male',
645: 'typically',
646: 'mass',
647: 'complex',
648: 'lower',
649: 'includes',
650: 'outside',
651: 'legal',
652: 'complete',
653: 'parliament',
654: 'significant',
655: 'actually',
656: 'business',
657: 'fiction',
658: 'physical',
659: 'followed',
660: 'deaths',
661: 'key',
662: 'leader',
663: 'widely',
664: 'page',
665: 'basic',
666: 'types',
667: 'henry',
668: 'beginning',
669: 'elected',
670: 'fire',
671: 'building',
672: 'independence',
673: 'went',
674: 'movie',
675: 'aircraft',
676: 'ever',
677: 'canadian',
678: 'material',
679: 'births',
680: 'video',
681: 'news',
682: 'future',
683: 'scientific',
684: 'simply',
685: 'go',
686: 'defined',
687: 'laws',
688: 'get',
689: 'close',
690: 'industry',
691: 'specific',
692: 'examples',
693: 'services',
694: 'believe',
695: 'idea',
696: 'introduced',
697: 'method',
698: 'points',
699: 'return',
700: 'cause',
701: 'indian',
702: 'britain',
703: 'features',
704: 'majority',
705: 'size',
706: 'post',
707: 'lead',
708: 'organization',
709: 'cannot',
710: 'designed',
711: 'ireland',
712: 'cross',
713: 'classical',
714: 'personal',
715: 'writing',
716: 'concept',
717: 'associated',
718: 'required',
719: 'soon',
720: 'changes',
721: 'located',
722: 'california',
723: 'sense',
724: 'believed',
725: 'away',
726: 'started',
727: 'co',
728: 'religion',
729: 'mother',
730: 'county',
731: 'rules',
732: 'studies',
733: 'yet',
734: 'find',
735: 'knowledge',
736: 'put',
737: 'founded',
738: 'policy',
739: 'currently',
740: 'provide',
741: 'working',
742: 'media',
743: 'election',
744: 'australian',
745: 'me',
746: 'thomas',
747: 'allowed',
748: 'russia',
749: 'greater',
750: 'earlier',
751: 'limited',
752: 'object',
753: 'brought',
754: 'online',
755: 'lord',
756: 'association',
757: 'mostly',
758: 'blue',
759: 'across',
760: 'constitution',
761: 'added',
762: 'interest',
763: 'things',
764: 'relations',
765: 'speed',
766: 'federal',
767: 'singer',
768: 'effects',
769: 'growth',
770: 'sources',
771: 'your',
772: 'remains',
773: 'z',
774: 'probably',
775: 'gave',
776: 'simple',
777: 'attack',
778: 'longer',
779: 'reference',
780: 'saint',
781: 'success',
782: 'killed',
783: 'past',
784: 'career',
785: 'need',
786: 'park',
787: 'definition',
788: 'say',
789: 'etc',
790: 'peace',
791: 'give',
792: 'chief',
793: 'stories',
794: 'security',
795: 'wide',
796: 'ball',
797: 'saw',
798: 'machine',
799: 'better',
800: 'cell',
801: 'leading',
802: 'becomes',
803: 'spain',
804: 'larger',
805: 'products',
806: 'night',
807: 'parties',
808: 'remained',
809: 'prize',
810: 'website',
811: 'big',
812: 'months',
813: 'money',
814: 'cultural',
815: 'help',
816: 'territory',
817: 'private',
818: 'moved',
819: 'wife',
820: 'letter',
821: 'lines',
822: 'politics',
823: 'largely',
824: 'contains',
825: 'companies',
826: 'lake',
827: 'perhaps',
828: 'green',
829: 'already',
830: 'dead',
831: 'iii',
832: 'library',
833: 'separate',
834: 'refer',
835: 'makes',
836: 'appeared',
837: 'dutch',
838: 'holy',
839: 'era',
840: 'novel',
841: 'successful',
842: 'italy',
843: 'letters',
844: 'results',
845: 'matter',
846: 'produce',
847: 'origin',
848: 'claim',
849: 'whole',
850: 'attempt',
851: 'directly',
852: 'actress',
853: 'surface',
854: 'revolution',
855: 'highly',
856: 'caused',
857: 'status',
858: 'musical',
859: 'richard',
860: 'commercial',
861: 'division',
862: 'color',
863: 'health',
864: 'coast',
865: 'release',
866: 'latter',
867: 'authority',
868: 'treaty',
869: 'turn',
870: 'michael',
871: 'nation',
872: 'direct',
873: 'asia',
874: 'edition',
875: 'programming',
876: 'playing',
877: 'date',
878: 'whom',
879: 'mary',
880: 'native',
881: 'married',
882: 'towards',
883: 'issues',
884: 'double',
885: 'primary',
886: 'basis',
887: 'allow',
888: 'enough',
889: 'memory',
890: 'reason',
891: 'web',
892: 'exist',
893: 'provided',
894: 'oil',
895: 'course',
896: 'functions',
897: 'chemical',
898: 'alexander',
899: 'analysis',
900: 'replaced',
901: 'mid',
902: 'queen',
903: 'tv',
904: 'claims',
905: 'sun',
906: 'literature',
907: 'metal',
908: 'amount',
909: 'divided',
910: 'blood',
911: 'likely',
912: 'access',
913: 'average',
914: 'length',
915: 'smaller',
916: 'medical',
917: 'property',
918: 'students',
919: 'degree',
920: 'elections',
921: 'club',
922: 'claimed',
923: 'performance',
924: 'director',
925: 'digital',
926: 'museum',
927: 'front',
928: 'difficult',
929: 'tradition',
930: 'nearly',
931: 'schools',
932: 'washington',
933: 'gas',
934: 'map',
935: 'jesus',
936: 'rome',
937: 'louis',
938: 'unit',
939: 'baseball',
940: 'mind',
941: 'mark',
942: 'peter',
943: 'collection',
944: 'product',
945: 'congress',
946: 'programs',
947: 'changed',
948: 'ideas',
949: 'moon',
950: 'entire',
951: 'user',
952: 'ground',
953: 'records',
954: 'frequently',
955: 'increase',
956: 'highest',
957: 'finally',
958: 'sent',
959: 'board',
960: 'don',
961: 'notable',
962: 'methods',
963: 'read',
964: 'recently',
965: 'bit',
966: 'involved',
967: 'variety',
968: 'call',
969: 'democratic',
970: 'ten',
971: 'served',
972: 'minor',
973: 'hard',
974: 'birth',
975: 'objects',
976: 'increased',
977: 'nuclear',
978: 'section',
979: 'street',
980: 'windows',
981: 'relatively',
982: 'car',
983: 'move',
984: 'create',
985: 'returned',
986: 'bank',
987: 'conditions',
988: 'operation',
989: 'adopted',
990: 'relationship',
991: 'christ',
992: 'hall',
993: 'appear',
994: 'rest',
995: 'child',
996: 'element',
997: 'appears',
998: 'takes',
999: 'fall',
...}
In [8]:
int_words
Out[8]:
[5243,
3081,
11,
5,
194,
1,
3136,
45,
58,
155,
127,
741,
476,
10580,
133,
0,
27670,
1,
0,
102,
854,
2,
0,
15152,
58493,
1,
0,
150,
854,
3584,
0,
194,
10,
190,
58,
4,
5,
10760,
214,
6,
1325,
104,
454,
19,
58,
2732,
362,
6,
3675,
0,
708,
1,
371,
26,
40,
36,
53,
540,
97,
11,
5,
1423,
2758,
18,
567,
686,
7115,
0,
247,
5243,
10,
1052,
27,
0,
320,
248,
45380,
2879,
792,
186,
5243,
11,
5,
200,
602,
10,
0,
1135,
19,
2622,
25,
9023,
2,
279,
31,
4148,
141,
59,
25,
6442,
4197,
1,
153,
32,
362,
5243,
36,
1137,
6,
447,
345,
1818,
19,
4862,
0,
6757,
1,
7597,
1775,
566,
0,
93,
0,
247,
11100,
11,
51,
7115,
89,
26,
270,
37,
5958,
4865,
20509,
28,
56591,
41,
317,
5,
25910,
527,
7597,
371,
4,
258,
1,
153,
25,
1206,
11,
7597,
200,
1579,
2,
15215,
332,
1775,
7115,
4862,
345,
764,
160,
406,
5691,
756,
1,
4115,
1131,
4333,
1536,
2,
567,
8137,
98,
5243,
10,
51,
1407,
686,
18,
153,
26,
10,
155,
7115,
36,
2035,
1423,
8177,
1,
153,
46,
694,
6,
31,
5,
4158,
246,
371,
76,
948,
78,
310,
30,
4779,
371,
508,
139,
2312,
3560,
364,
23,
1822,
6,
1902,
59,
10,
36,
8412,
78,
310,
5,
246,
371,
508,
31,
753,
78,
1734,
2,
8041,
24533,
2,
275,
1693,
19,
151,
1033,
95,
224,
371,
17,
1817,
24,
4779,
1556,
51,
8144,
1467,
24533,
2,
12775,
4,
6130,
19,
4189,
21262,
2429,
39,
16478,
2,
7244,
861,
1,
1194,
10210,
2515,
28,
15190,
187,
2,
48,
1123,
912,
6,
1049,
467,
12323,
7115,
133,
0,
0,
11100,
3061,
2,
12131,
734,
4779,
6600,
4,
14652,
27,
450,
485,
24533,
198,
295,
948,
4,
20095,
20358,
1,
54250,
300,
6,
24533,
20358,
16614,
0,
30651,
1,
0,
93,
44,
3837,
2,
57262,
2,
3813,
0,
3413,
1,
0,
1777,
187,
1,
0,
626,
0,
13245,
1,
3,
21,
90,
122,
284,
25,
243,
232,
6,
31,
436,
26381,
1,
173,
5243,
7727,
2534,
4,
29,
95,
1,
289,
602,
4316,
19,
0,
13245,
16614,
49,
187,
117,
46,
359,
19,
0,
384,
242,
96,
31,
5242,
34,
331,
2723,
18,
0,
838,
1378,
27,
32,
8776,
46,
6473,
34,
3246,
0,
27670,
28,
551,
43318,
39,
30,
127,
54091,
435,
84,
0,
64,
1,
0,
102,
462,
83,
2,
25,
232,
18,
47,
11,
26381,
1,
173,
5243,
4,
0,
173,
839,
0,
45,
6,
89,
0,
194,
6,
1014,
1043,
42,
69,
4865,
17,
937,
15016,
4576,
164,
4,
29,
40814,
11839,
23262,
259,
1225,
59760,
3,
22,
7,
16,
99,
35,
628,
0,
2662,
63,
371,
33,
48,
75,
93,
687,
14848,
4050,
28,
817,
917,
11,
101,
4,
11100,
2534,
362,
5,
6060,
2,
662,
4,
0,
63,
701,
435,
40,
4224,
1289,
19,
35,
10,
30,
4779,
2,
94,
25,
49,
29,
5227,
4,
3,
22,
8,
16,
4,
0,
5359,
1,
0,
150,
854,
467,
12323,
334,
30,
17347,
2434,
200,
1293,
141,
12323,
215,
37,
89,
0,
247,
5243,
55,
116,
7115,
38,
1206,
32,
197,
11,
0,
45,
181,
4779,
529,
2,
12323,
11,
0,
1834,
1,
1914,
5243,
41,
34,
32,
228,
75,
4779,
435,
733,
2404,
2,
0,
194,
17,
86,
1073,
11,
30,
14143,
30818,
18,
0,
12080,
60733,
34,
50,
2422,
616,
4,
0,
150,
854,
0,
45,
567,
8756,
4779,
2937,
1034,
14592,
26,
10,
636,
359,
19,
26,
5385,
154,
191,
2937,
1034,
14592,
334,
153,
10,
917,
4,
3,
12,
20,
7,
19,
0,
194,
4779,
17,
989,
11,
5,
567,
1607,
26,
10,
13,
32,
890,
19,
47,
848,
14592,
11,
0,
1834,
1,
173,
4779,
208,
4,
153,
10,
917,
14592,
5469,
23,
0,
419,
12993,
917,
10,
6892,
4,
32,
139,
35,
1346,
0,
2877,
1,
15190,
917,
154,
99,
3470,
38,
652,
445,
6,
89,
2,
3136,
43,
917,
11,
46,
3597,
54,
11,
15443,
1555,
13,
2966,
4,
44,
258,
14592,
1164,
153,
35,
88,
3223,
1131,
52,
38,
751,
445,
6,
89,
1049,
522,
2,
1791,
4,
5806,
23,
1556,
1,
5128,
2,
1293,
14592,
14,
2655,
1,
11100,
33,
35,
88,
30074,
966,
30,
1334,
518,
99,
1131,
2,
308,
159,
512,
0,
805,
1,
43,
1258,
218,
1258,
1037,
33,
1391,
0,
908,
1,
741,
64,
966,
4,
442,
32,
73,
2917,
19,
75,
3,
73,
2966,
27,
0,
1258,
1,
275,
1555,
159,
4325,
2520,
498,
4,
727,
10639,
18856,
30,
762,
246,
986,
73,
31,
177,
97,
6,
740,
3275,
23,
912,
6,
0,
362,
1,
442,
14592,
14,
948,
39,
2025,
222,
150,
741,
476,
1818,
2,
29,
2987,
39,
1022,
4,
0,
854,
1,
3,
12,
20,
12,
4,
302,
14592,
14,
602,
1,
917,
10,
647,
26,
17,
347,
4,
5,
112,
1,
311,
79,
29,
3572,
2,
59,
25,
6442,
4197,
1,
47,
1,
29,
948,
13,
50,
2158,
1591,
67,
621,
2752,
26722,
14,
15787,
4,
29,
0,
10468,
2,
44,
199,
26722,
1460,
19,
51,
636,
1045,
345,
1775,
133,
0,
2581,
1,
93,
917,
11,
5,
244,
389,
445,
4,
152,
2,
0,
147,
2581,
1,
371,
39,
4593,
20895,
28,
9219,
4,
0,
940,
1974,
1,
371,
19,
0,
1131,
25,
44,
1728,
35,
5216,
15787,
2,
5,
143,
1,
4,
33,
1131,
73,
9764,
4,
5826,
1,
53601,
66,
61,
26,
17,
4,
43,
567,
762,
6,
175,
94,
13,
118,
917,
684,
1146,
78,
123,
508,
13102,
5574,
310,
6,
369,
6,
4107,
0,
1797,
6,
118,
5464,
917,
2,
153,
71,
38,
4,
608,
183,
19,
10,
608,
199,
94,
184,
11,
71,
6189,
9038,
11,
7205,
71,
1225,
0,
34192,
1,
0,
1797,
26722,
363,
88,
391,
30,
4779,
35,
1045,
66,
0,
2758,
43507,
2070,
29,
948,
39,
2025,
24,
55,
9944,
7115,
141,
4197,
1,
29,
501,
25,
3236,
63,
10326,
5243,
3245,
9614,
4,
3,
...]
Words that show up often such as "the", "of", and "for" don't provide much context to the nearby words. If we discard some of them, we can remove some of the noise from our data and in return get faster training and better representations. This process is called subsampling by Mikolov. For each word $w_i$ in the training set, we'll discard it with probability given by
$$ P(w_i) = 1 - \sqrt{\frac{t}{f(w_i)}} $$where $t$ is a threshold parameter and $f(w_i)$ is the frequency of word $w_i$ in the total dataset.
I'm going to leave this up to you as an exercise. This is more of a programming challenge, than about deep learning specifically. But, being able to prepare your data for your network is an important skill to have. Check out my solution to see how I did it.
Exercise: Implement subsampling for the words in
int_words. That is, go throughint_wordsand discard each word given the probablility $P(w_i)$ shown above. Note that $P(w_i)$ is the probability that a word is discarded. Assign the subsampled data totrain_words.
In [9]:
## Your code here
from collections import Counter
trhd = 1e-5
word_counts = Counter(int_words)
total_count = len(int_words)
In [10]:
freqs = {word: count / total_count for word, count in word_counts.items()}
In [11]:
p_drop = {word: 1 - np.sqrt(trhd / freq) for word, freq in freqs.items()}
In [12]:
# The final subsampled word list
import random
train_words = [word for word in int_words if p_drop[word] < random.random()]
In [13]:
len(train_words)
Out[13]:
4626922
In [14]:
train_words
Out[14]:
[5243,
194,
3136,
58,
476,
10580,
27670,
102,
854,
15152,
58493,
150,
3584,
10760,
362,
371,
540,
1423,
7115,
45380,
0,
1135,
9023,
4148,
6442,
4197,
153,
5243,
1137,
4862,
6757,
7597,
247,
11100,
7115,
5958,
4865,
20509,
56591,
317,
25910,
7597,
371,
7597,
200,
15215,
7115,
4862,
406,
5691,
4115,
4333,
1536,
8137,
5243,
1407,
155,
7115,
2035,
8177,
4158,
78,
4779,
1822,
8412,
8041,
24533,
1693,
19,
1033,
1556,
51,
8144,
1467,
24533,
12775,
6130,
4189,
21262,
2429,
16478,
2,
7244,
10210,
15190,
1123,
1049,
467,
12323,
7115,
133,
11100,
12131,
4779,
6600,
14652,
450,
24533,
20095,
20358,
54250,
24533,
20358,
16614,
30651,
3837,
57262,
3413,
1777,
187,
13245,
436,
26381,
5243,
7727,
19,
13245,
16614,
34,
331,
2723,
8776,
6473,
3246,
27670,
551,
43318,
54091,
84,
26381,
5243,
4865,
15016,
4576,
40814,
11839,
23262,
1225,
59760,
371,
14848,
4050,
817,
11100,
6060,
4779,
5359,
12323,
17347,
2434,
1293,
12323,
5243,
55,
7115,
12323,
4779,
0,
194,
1073,
30,
14143,
30818,
12080,
60733,
567,
8756,
4779,
14592,
5385,
14592,
917,
4,
1607,
14592,
4779,
4,
917,
14592,
5469,
12993,
6892,
1346,
2877,
15190,
3470,
89,
54,
15443,
2966,
14592,
5806,
5128,
2,
14592,
2655,
11100,
30074,
1131,
1258,
1037,
966,
4,
2917,
2966,
1258,
1555,
4325,
2520,
727,
10639,
18856,
762,
3275,
23,
14592,
948,
39,
2987,
14592,
3572,
6442,
29,
621,
2752,
26722,
15787,
10468,
44,
26722,
1460,
1045,
2581,
244,
20895,
9219,
15787,
9764,
5826,
53601,
917,
1146,
13102,
5574,
4107,
5464,
153,
199,
6189,
9038,
7205,
34192,
26722,
1045,
43507,
2070,
29,
2025,
9944,
4197,
29,
3236,
10326,
9614,
12,
10880,
5865,
5445,
54194,
2410,
7428,
5108,
17374,
1217,
7532,
3413,
1130,
817,
5865,
7445,
18901,
4779,
5762,
15304,
253,
3,
5865,
5639,
16467,
3245,
9614,
1589,
5865,
47088,
16690,
1707,
5243,
6528,
3245,
9614,
3416,
5243,
10880,
5865,
467,
12502,
2,
334,
3137,
663,
7095,
10326,
4779,
16467,
2061,
4,
9614,
4548,
10326,
5243,
6292,
12502,
4333,
4702,
948,
1619,
30928,
967,
1194,
14592,
606,
26722,
15787,
3382,
7308,
9614,
606,
2573,
4158,
91,
8622,
1708,
6241,
6087,
1555,
21549,
26210,
24286,
2357,
9231,
4090,
193,
6788,
19109,
20,
6149,
9678,
1673,
116,
4,
62687,
756,
88,
6783,
5243,
44,
6709,
1022,
1818,
2771,
10943,
481,
0,
8952,
1346,
3246,
33655,
1860,
6788,
19109,
2987,
1232,
193,
42566,
3281,
19109,
33392,
77,
3281,
2281,
7115,
210,
19109,
8316,
24533,
872,
11438,
248,
18049,
2579,
16558,
2987,
3228,
19109,
8802,
2981,
948,
7597,
4906,
5,
4987,
1250,
97,
1871,
476,
2239,
54978,
11663,
3079,
7115,
11962,
6060,
6783,
4369,
4630,
28641,
3246,
24533,
14592,
19109,
3246,
19680,
33655,
7115,
725,
19109,
2855,
21398,
8414,
4049,
724,
442,
6165,
2370,
4779,
1325,
391,
6060,
1320,
14592,
944,
110,
1258,
5478,
10608,
3543,
43,
948,
942,
24533,
240,
11831,
9689,
4684,
1312,
7871,
18856,
1966,
7499,
4398,
4333,
1536,
1992,
1159,
4779,
4049,
8205,
11003,
898,
30138,
5548,
29250,
3500,
48730,
546,
30198,
6813,
13326,
10840,
3246,
23,
142,
10326,
7115,
2301,
7775,
917,
3283,
23,
135,
9614,
3584,
36862,
24533,
5243,
8421,
5243,
4384,
13447,
13347,
4862,
7115,
3779,
133,
6994,
15888,
39430,
5109,
18,
7115,
11522,
328,
1669,
10086,
15888,
2667,
5243,
4384,
13447,
13347,
4343,
3575,
30707,
10100,
9862,
15414,
17635,
589,
13106,
3177,
5109,
13264,
17564,
59,
4065,
9003,
5362,
152,
6788,
19109,
46302,
40412,
135,
556,
1177,
7905,
5365,
8253,
1189,
5109,
40412,
19109,
61,
17885,
7115,
13860,
7115,
5216,
23411,
3375,
11715,
5243,
14525,
21099,
411,
1946,
7115,
717,
25088,
25088,
122,
476,
7338,
2749,
1023,
5478,
1466,
371,
5651,
1166,
1522,
54,
2281,
5548,
31515,
4779,
25088,
10932,
62085,
13063,
4115,
1555,
308,
3991,
25308,
15914,
21957,
61248,
55056,
11,
5548,
29250,
13702,
13702,
0,
55056,
6060,
3246,
2532,
60,
3,
55056,
818,
5548,
25088,
2545,
26330,
25088,
435,
803,
1555,
38943,
15567,
29773,
2125,
2532,
16297,
16889,
4779,
1817,
1009,
37858,
1857,
15549,
31515,
30198,
4384,
6931,
154,
572,
1595,
1271,
4328,
154,
887,
3203,
623,
1173,
20593,
51936,
10267,
54419,
4617,
30198,
737,
1294,
30198,
339,
16,
30198,
441,
5243,
4,
31515,
11567,
16929,
19092,
39,
661,
5243,
5028,
32566,
3111,
3895,
2332,
30198,
1555,
29250,
193,
25088,
1772,
7,
404,
5845,
803,
51936,
2241,
54419,
30198,
55056,
2862,
42,
29250,
1818,
211,
1555,
10763,
10763,
2474,
1895,
18045,
1555,
1451,
3439,
1297,
5548,
25088,
1895,
29773,
2,
5526,
1380,
11998,
848,
5548,
25088,
4343,
13576,
14844,
5548,
31515,
152,
18873,
1555,
10763,
872,
623,
854,
18693,
5243,
7115,
8873,
9678,
8448,
2483,
8873,
719,
7115,
9758,
43781,
7115,
2950,
2486,
8833,
8873,
4509,
7115,
83,
8731,
8873,
8199,
336,
280,
18,
26711,
51448,
63,
7115,
8205,
11003,
898,
30138,
748,
45121,
8448,
7994,
43781,
4773,
3507,
11622,
14314,
1728,
8448,
19109,
7941,
4987,
49,
1507,
8873,
854,
1694,
4779,
1555,
5820,
8448,
781,
2078,
5754,
5243,
1818,
29250,
55056,
38943,
46606,
571,
725,
5243,
1054,
4779,
14760,
26711,
51448,
3711,
7115,
1262,
3061,
26330,
6813,
27198,
2801,
6060,
4049,
7115,
2801,
14139,
1273,
4779,
4779,
210,
1556,
6483,
3203,
3060,
16300,
1555,
10763,
4779,
192,
2474,
4779,
4049,
7522,
81,
2,
10089,
5450,
30198,
5115,
6740,
18748,
7871,
5545,
5243,
39,
4960,
1074,
5450,
7115,
6990,
2520,
14942,
19666,
280,
4049,
176,
4754,
1977,
11095,
541,
5450,
2815,
161,
5160,
5450,
5450,
10,
7597,
30179,
20858,
38610,
208,
867,
18748,
689,
12665,
6994,
7115,
30198,
234,
3265,
2007,
44699,
30198,
2029,
1534,
30198,
4779,
2862,
1642,
476,
5144,
1649,
2483,
10314,
3111,
4779,
1615,
435,
6181,
1414,
12707,
377,
1,
46,
49035,
307,
5939,
4754,
7115,
3219,
952,
5398,
30198,
3117,
909,
14425,
4053,
16184,
6869,
32243,
10969,
16329,
11962,
7115,
7115,
966,
1572,
4754,
323,
7115,
8330,
527,
4754,
4987,
5216,
23656,
18748,
305,
10596,
328,
4123,
40,
347,
5243,
9579,
5243,
3375,
11715,
3828,
8440,
1700,
9043,
2815,
17069,
6723,
4197,
1,
5243,
7115,
212,
6609,
13206,
867,
46265,
41,
15302,
1169,
1372,
728,
13469,
869,
13735,
36589,
4862,
5243,
3375,
11715,
351,
212,
5,
12395,
21099,
7115,
1582,
2869,
342,
7115,
10260,
30466,
11100,
...]
Now that our data is in good shape, we need to get it into the proper form to pass it into our network. With the skip-gram architecture, for each word in the text, we want to grab all the words in a window around that word, with size $C$.
From Mikolov et al.:
"Since the more distant words are usually less related to the current word than those close to it, we give less weight to the distant words by sampling less from those words in our training examples... If we choose $C = 5$, for each training word we will select randomly a number $R$ in range $< 1; C >$, and then use $R$ words from history and $R$ words from the future of the current word as correct labels."
Exercise: Implement a function
get_targetthat receives a list of words, an index, and a window size, then returns a list of words in the window around the index. Make sure to use the algorithm described above, where you choose a random number of words from the window.
In [15]:
def get_target(words, idx, window_size=5):
''' Get a list of words in a window around an index. '''
# Your code here
# select r before and after
R = np.random.randint(1, window_size + 1)
start = idx - R if (idx - R) > 0 else 0
stop = idx + R # todo, find out the max index for the words
targetWords = set(words[start:idx] + words[idx + 1: stop + 1])
return list(targetWords)
Here's a function that returns batches for our network. The idea is that it grabs batch_size words from a words list. Then for each of those words, it gets the target words in the window. I haven't found a way to pass in a random number of target words and get it to work with the architecture, so I make one row per input-target pair. This is a generator function by the way, helps save memory.
In [16]:
def get_batches(words, batch_size, window_size=5):
''' Create a generator of word batches as a tuple (inputs, targets) '''
n_batches = len(words)//batch_size
# only full batches
words = words[:n_batches*batch_size]
for idx in range(0, len(words), batch_size):
x, y = [], []
batch = words[idx:idx+batch_size]
for ii in range(len(batch)):
batch_x = batch[ii]
batch_y = get_target(batch, ii, window_size)
y.extend(batch_y)
x.extend([batch_x]*len(batch_y))
yield x, y
From Chris McCormick's blog, we can see the general structure of our network.
The input words are passed in as one-hot encoded vectors. This will go into a hidden layer of linear units, then into a softmax layer. We'll use the softmax layer to make a prediction like normal.
The idea here is to train the hidden layer weight matrix to find efficient representations for our words. This weight matrix is usually called the embedding matrix or embedding look-up table. We can discard the softmax layer becuase we don't really care about making predictions with this network. We just want the embedding matrix so we can use it in other networks we build from the dataset.
I'm going to have you build the graph in stages now. First off, creating the inputs and labels placeholders like normal.
Exercise: Assign
inputsandlabelsusingtf.placeholder. We're going to be passing in integers, so set the data types totf.int32. The batches we're passing in will have varying sizes, so set the batch sizes to [None]. To make things work later, you'll need to set the second dimension oflabelstoNoneor1.
In [17]:
train_graph = tf.Graph()
with train_graph.as_default():
inputs = tf.placeholder(tf.int32, [None], name='inputs')
labels = tf.placeholder(tf.int32, [None, None], name='labels')
The embedding matrix has a size of the number of words by the number of neurons in the hidden layer. So, if you have 10,000 words and 300 hidden units, the matrix will have size $10,000 \times 300$. Remember that we're using one-hot encoded vectors for our inputs. When you do the matrix multiplication of the one-hot vector with the embedding matrix, you end up selecting only one row out of the entire matrix:
You don't actually need to do the matrix multiplication, you just need to select the row in the embedding matrix that corresponds to the input word. Then, the embedding matrix becomes a lookup table, you're looking up a vector the size of the hidden layer that represents the input word.
Exercise: Tensorflow provides a convenient function
tf.nn.embedding_lookupthat does this lookup for us. You pass in the embedding matrix and a tensor of integers, then it returns rows in the matrix corresponding to those integers. Below, set the number of embedding features you'll use (200 is a good start), create the embedding matrix variable, and usetf.nn.embedding_lookupto get the embedding tensors. For the embedding matrix, I suggest you initialize it with a uniform random numbers between -1 and 1 using tf.random_uniform. This TensorFlow tutorial will help if you get stuck.
In [18]:
n_vocab = len(int_to_vocab)
n_embedding = 300 # Number of embedding features
with train_graph.as_default():
# create embedding weight matrix here
embedding = tf.Variable(tf.random_uniform((n_vocab, n_embedding), -1, 1))
# use tf.nn.embedding_lookup to get the hidden layer output
embed = tf.nn.embedding_lookup(embedding, inputs)
For every example we give the network, we train it using the output from the softmax layer. That means for each input, we're making very small changes to millions of weights even though we only have one true example. This makes training the network very inefficient. We can approximate the loss from the softmax layer by only updating a small subset of all the weights at once. We'll update the weights for the correct label, but only a small number of incorrect labels. This is called "negative sampling". Tensorflow has a convenient function to do this, tf.nn.sampled_softmax_loss.
Exercise: Below, create weights and biases for the softmax layer. Then, use
tf.nn.sampled_softmax_lossto calculate the loss. Be sure to read the documentation to figure out how it works.
In [19]:
# Number of negative labels to sample
n_sampled = 100
with train_graph.as_default():
# create softmax weight matrix here
softmax_w = tf.Variable(tf.truncated_normal((n_vocab, n_embedding), stddev=0.1))
# create softmax biases here
softmax_b = tf.Variable(tf.zeros(n_vocab))
# Calculate the loss using negative sampling
loss = tf.nn.sampled_softmax_loss(softmax_w, softmax_b,
labels,
embed,
n_sampled,
n_vocab)
cost = tf.reduce_mean(loss)
optimizer = tf.train.AdamOptimizer().minimize(cost)
In [20]:
with train_graph.as_default():
## From Thushan Ganegedara's implementation
valid_size = 16 # Random set of words to evaluate similarity on.
valid_window = 100
# pick 8 samples from (0,100) and (1000,1100) each ranges. lower id implies more frequent
valid_examples = np.array(random.sample(range(valid_window), valid_size//2))
valid_examples = np.append(valid_examples,
random.sample(range(1000,1000+valid_window), valid_size//2))
valid_dataset = tf.constant(valid_examples, dtype=tf.int32)
# We use the cosine distance:
norm = tf.sqrt(tf.reduce_sum(tf.square(embedding), 1, keep_dims=True))
normalized_embedding = embedding / norm
valid_embedding = tf.nn.embedding_lookup(normalized_embedding, valid_dataset)
similarity = tf.matmul(valid_embedding, tf.transpose(normalized_embedding))
In [21]:
# If the checkpoints directory doesn't exist:
!mkdir checkpoints
In [22]:
epochs = 10
batch_size = 1000
window_size = 10
with train_graph.as_default():
saver = tf.train.Saver()
with tf.Session(graph=train_graph) as sess:
iteration = 1
loss = 0
sess.run(tf.global_variables_initializer())
for e in range(1, epochs+1):
batches = get_batches(train_words, batch_size, window_size)
start = time.time()
for x, y in batches:
feed = {inputs: x,
labels: np.array(y)[:, None]}
train_loss, _ = sess.run([cost, optimizer], feed_dict=feed)
loss += train_loss
if iteration % 100 == 0:
end = time.time()
print("Epoch {}/{}".format(e, epochs),
"Iteration: {}".format(iteration),
"Avg. Training loss: {:.4f}".format(loss/100),
"{:.4f} sec/batch".format((end-start)/100))
loss = 0
start = time.time()
if iteration % 1000 == 0:
## From Thushan Ganegedara's implementation
# note that this is expensive (~20% slowdown if computed every 500 steps)
sim = similarity.eval()
for i in range(valid_size):
valid_word = int_to_vocab[valid_examples[i]]
top_k = 8 # number of nearest neighbors
nearest = (-sim[i, :]).argsort()[1:top_k+1]
log = 'Nearest to %s:' % valid_word
for k in range(top_k):
close_word = int_to_vocab[nearest[k]]
log = '%s %s,' % (log, close_word)
print(log)
iteration += 1
save_path = saver.save(sess, "checkpoints/text8.ckpt")
embed_mat = sess.run(normalized_embedding)
Epoch 1/10 Iteration: 100 Avg. Training loss: 5.7525 0.2803 sec/batch
Epoch 1/10 Iteration: 200 Avg. Training loss: 5.7441 0.2646 sec/batch
Epoch 1/10 Iteration: 300 Avg. Training loss: 5.5991 0.2583 sec/batch
Epoch 1/10 Iteration: 400 Avg. Training loss: 5.6885 0.2514 sec/batch
Epoch 1/10 Iteration: 500 Avg. Training loss: 5.6569 0.2449 sec/batch
Epoch 1/10 Iteration: 600 Avg. Training loss: 5.6666 0.2403 sec/batch
Epoch 1/10 Iteration: 700 Avg. Training loss: 5.6746 0.2341 sec/batch
Epoch 1/10 Iteration: 800 Avg. Training loss: 5.6371 0.2338 sec/batch
Epoch 1/10 Iteration: 900 Avg. Training loss: 5.5736 0.2318 sec/batch
Epoch 1/10 Iteration: 1000 Avg. Training loss: 5.5347 0.2329 sec/batch
Nearest to time: lowe, yage, niqqud, alf, jj, absorbent, mears, obstetrics,
Nearest to his: flags, ippon, melanogaster, plays, preponderance, traveler, choking, numbing,
Nearest to on: prut, job, hour, cleric, endowments, yucatan, lle, durians,
Nearest to many: timeline, presidente, component, kovacs, nestor, maskhadov, vesicle, davy,
Nearest to was: ascertaining, bulging, aliyah, wolfgang, clashing, mandaean, bfbs, mil,
Nearest to that: arnage, repealed, isospin, anzac, borromini, tupper, scoffed, ansgar,
Nearest to states: deaths, determinants, ocogs, shafts, children, wanderer, interpreters, superconducting,
Nearest to known: ldf, retaking, enabling, breakfasts, ulm, porter, coru, harmless,
Nearest to active: sculptors, facilitated, ton, bcc, daddy, perishable, yugoslavian, amesbury,
Nearest to existence: wheatstone, olcott, aging, fares, despicable, afi, archipel, nauvoo,
Nearest to hold: arbitration, concacaf, ansar, byng, holidays, nanobots, darin, congruences,
Nearest to pressure: anfa, reliefs, conference, venerate, politiken, wedged, pools, fansubs,
Nearest to bible: garrisoned, monologues, procure, friendly, marzipan, tocantins, everson, mullis,
Nearest to joseph: nickel, csma, ancient, usk, otello, harring, colonies, puente,
Nearest to gold: resorted, patently, plebs, monarchist, shingles, endosymbionts, oda, loft,
Nearest to except: thyroxine, demobilized, journal, protective, vindication, madame, recruiting, outranks,
Epoch 1/10 Iteration: 1100 Avg. Training loss: 5.5728 0.2337 sec/batch
Epoch 1/10 Iteration: 1200 Avg. Training loss: 5.4228 0.2314 sec/batch
Epoch 1/10 Iteration: 1300 Avg. Training loss: 5.4065 0.2294 sec/batch
Epoch 1/10 Iteration: 1400 Avg. Training loss: 5.2999 0.2297 sec/batch
Epoch 1/10 Iteration: 1500 Avg. Training loss: 5.2622 0.2289 sec/batch
Epoch 1/10 Iteration: 1600 Avg. Training loss: 5.2392 0.2314 sec/batch
Epoch 1/10 Iteration: 1700 Avg. Training loss: 5.1487 0.2319 sec/batch
Epoch 1/10 Iteration: 1800 Avg. Training loss: 5.1344 0.2307 sec/batch
Epoch 1/10 Iteration: 1900 Avg. Training loss: 5.0453 0.2301 sec/batch
Epoch 1/10 Iteration: 2000 Avg. Training loss: 5.0415 0.2297 sec/batch
Nearest to time: face, absorbent, lowe, political, because, mears, conclusions, substance,
Nearest to his: flags, plays, melanogaster, davenport, ippon, preponderance, choking, traveler,
Nearest to on: job, hour, endowments, referred, cleric, republica, yucatan, prut,
Nearest to many: timeline, component, presidente, davy, merely, kovacs, dancer, nestor,
Nearest to was: ascertaining, bulging, economics, wolfgang, venezuela, nik, though, martha,
Nearest to that: arnage, shares, repealed, relative, isospin, anzac, borromini, committed,
Nearest to states: children, deaths, shafts, refined, ocogs, determinants, wanderer, interpreters,
Nearest to known: enabling, retaking, ulm, ldf, porter, opening, harmless, pronouncing,
Nearest to active: sculptors, facilitated, speed, ton, bcc, yugoslavian, handed, daddy,
Nearest to existence: olcott, document, assumes, oldest, aging, exclude, bars, solstice,
Nearest to hold: concacaf, holidays, arbitration, academic, byng, asserts, entirely, unheard,
Nearest to pressure: anfa, conference, pools, conducted, druidic, normally, politiken, fansubs,
Nearest to bible: friendly, garrisoned, monologues, tocantins, procure, largely, insights, everson,
Nearest to joseph: nickel, ancient, csma, usk, cylinders, colonies, yeah, breadth,
Nearest to gold: resorted, mechanism, shingles, legislation, loft, oda, distance, monarchist,
Nearest to except: journal, demobilized, protective, madame, racer, happen, unifying, brace,
Epoch 1/10 Iteration: 2100 Avg. Training loss: 4.9776 0.2347 sec/batch
Epoch 1/10 Iteration: 2200 Avg. Training loss: 4.9819 0.2354 sec/batch
Epoch 1/10 Iteration: 2300 Avg. Training loss: 4.9411 0.2314 sec/batch
Epoch 1/10 Iteration: 2400 Avg. Training loss: 4.9143 0.2369 sec/batch
Epoch 1/10 Iteration: 2500 Avg. Training loss: 4.8908 0.2336 sec/batch
Epoch 1/10 Iteration: 2600 Avg. Training loss: 4.8782 0.2344 sec/batch
Epoch 1/10 Iteration: 2700 Avg. Training loss: 4.8430 0.2311 sec/batch
Epoch 1/10 Iteration: 2800 Avg. Training loss: 4.8585 0.2354 sec/batch
Epoch 1/10 Iteration: 2900 Avg. Training loss: 4.8430 0.2379 sec/batch
Epoch 1/10 Iteration: 3000 Avg. Training loss: 4.8468 0.2379 sec/batch
Nearest to time: lowe, alf, absorbent, jj, mears, niqqud, face, pasture,
Nearest to his: flags, plays, melanogaster, davenport, preponderance, ippon, boulevard, northernmost,
Nearest to on: job, hour, yucatan, endowments, cleric, republica, recalling, prut,
Nearest to many: presidente, component, kovacs, devastation, dancer, timeline, circular, merely,
Nearest to was: ascertaining, martha, mil, bulging, clashing, bfbs, eileen, greenhouse,
Nearest to that: arnage, repealed, shares, isospin, anzac, relative, appointment, tupper,
Nearest to states: children, deaths, ocogs, wanderer, structuralist, determinants, refined, shafts,
Nearest to known: ldf, enabling, retaking, ulm, porter, breakfasts, katherine, harmless,
Nearest to active: facilitated, ton, sculptors, daddy, speed, yugoslavian, bcc, competing,
Nearest to existence: aging, olcott, assumes, document, fares, despicable, strokes, wheatstone,
Nearest to hold: concacaf, arbitration, specify, byng, holidays, academic, stanzas, unheard,
Nearest to pressure: anfa, promise, wedged, conducted, politiken, druidic, pools, reactant,
Nearest to bible: garrisoned, friendly, procure, monologues, insights, tocantins, largely, internally,
Nearest to joseph: ancient, nickel, cylinders, colonies, usk, yeah, csma, breadth,
Nearest to gold: resorted, loft, discontinued, oda, monarchist, legislation, shingles, distance,
Nearest to except: journal, demobilized, thyroxine, racer, vindication, unifying, madame, happen,
Epoch 1/10 Iteration: 3100 Avg. Training loss: 4.8290 0.2398 sec/batch
Epoch 1/10 Iteration: 3200 Avg. Training loss: 4.8172 0.2331 sec/batch
Epoch 1/10 Iteration: 3300 Avg. Training loss: 4.7706 0.2345 sec/batch
Epoch 1/10 Iteration: 3400 Avg. Training loss: 4.7406 0.2339 sec/batch
Epoch 1/10 Iteration: 3500 Avg. Training loss: 4.7772 0.2338 sec/batch
Epoch 1/10 Iteration: 3600 Avg. Training loss: 4.7500 0.2322 sec/batch
Epoch 1/10 Iteration: 3700 Avg. Training loss: 4.7526 0.2333 sec/batch
Epoch 1/10 Iteration: 3800 Avg. Training loss: 4.7787 0.2321 sec/batch
Epoch 1/10 Iteration: 3900 Avg. Training loss: 4.7310 0.2318 sec/batch
Epoch 1/10 Iteration: 4000 Avg. Training loss: 4.7002 0.2301 sec/batch
Nearest to time: jefferson, lowe, alf, absorbent, jj, niqqud, yage, pasture,
Nearest to his: flags, ippon, plays, prodigy, davenport, wolfe, serfdom, preponderance,
Nearest to on: hour, endowments, job, recalling, cleric, yucatan, prut, yates,
Nearest to many: presidente, kovacs, devastation, davy, maskhadov, component, vesicle, herding,
Nearest to was: martha, ascertaining, mil, eileen, aliyah, clashing, bulging, hemophilia,
Nearest to that: arnage, isospin, repealed, anzac, shares, relative, borromini, appointment,
Nearest to states: children, ocogs, deaths, wanderer, illyrian, structuralist, papa, refined,
Nearest to known: ldf, retaking, enabling, porter, ulm, breakfasts, coru, katherine,
Nearest to active: ton, facilitated, sculptors, daddy, speed, yugoslavian, bcc, chopper,
Nearest to existence: aging, olcott, assumes, despicable, document, wheatstone, exclude, fares,
Nearest to hold: concacaf, arbitration, specify, byng, unheard, darin, academic, holidays,
Nearest to pressure: anfa, wedged, politiken, venerate, pools, promise, druidic, umayyads,
Nearest to bible: procure, insights, monologues, everson, garrisoned, tocantins, friendly, ethnicity,
Nearest to joseph: ancient, usk, colonies, nickel, paraguay, yeah, cylinders, puente,
Nearest to gold: resorted, loft, courbet, tisza, discontinued, shingles, patently, oda,
Nearest to except: demobilized, journal, thyroxine, vindication, racer, beamed, happen, unifying,
Epoch 1/10 Iteration: 4100 Avg. Training loss: 4.7244 0.2268 sec/batch
Epoch 1/10 Iteration: 4200 Avg. Training loss: 4.7089 0.2257 sec/batch
Epoch 1/10 Iteration: 4300 Avg. Training loss: 4.6224 0.2258 sec/batch
Epoch 1/10 Iteration: 4400 Avg. Training loss: 4.6442 0.2276 sec/batch
Epoch 1/10 Iteration: 4500 Avg. Training loss: 4.6729 0.2247 sec/batch
Epoch 1/10 Iteration: 4600 Avg. Training loss: 4.6608 0.2272 sec/batch
Epoch 2/10 Iteration: 4700 Avg. Training loss: 4.5884 0.1696 sec/batch
Epoch 2/10 Iteration: 4800 Avg. Training loss: 4.5369 0.2294 sec/batch
Epoch 2/10 Iteration: 4900 Avg. Training loss: 4.5137 0.2294 sec/batch
Epoch 2/10 Iteration: 5000 Avg. Training loss: 4.5009 0.2279 sec/batch
Nearest to time: jefferson, yage, absorbent, lowe, niqqud, mears, pasture, alf,
Nearest to his: flags, ippon, prodigy, wolfe, serfdom, davenport, lili, grandson,
Nearest to on: endowments, hour, job, cleric, yucatan, yates, prut, recalling,
Nearest to many: presidente, component, vesicle, kovacs, maskhadov, devastation, supplant, rexx,
Nearest to was: martha, ascertaining, aliyah, clashing, hemophilia, bfbs, bulging, eileen,
Nearest to that: arnage, isospin, repealed, scoffed, relative, borromini, shares, anzac,
Nearest to states: ocogs, children, illyrian, ambassador, deaths, refined, habitual, rallied,
Nearest to known: retaking, ldf, ulm, enabling, porter, breakfasts, coru, katherine,
Nearest to active: ton, facilitated, speed, daddy, bcc, yugoslavian, nami, sculptors,
Nearest to existence: aging, assumes, nauvoo, olcott, wheatstone, emphasize, document, what,
Nearest to hold: concacaf, arbitration, specify, unheard, academic, foil, byng, stanzas,
Nearest to pressure: anfa, wedged, venerate, politiken, normally, pools, umayyads, reactant,
Nearest to bible: procure, everson, observance, monologues, insights, tocantins, marzipan, garrisoned,
Nearest to joseph: ancient, usk, puente, colonies, paraguay, walter, ordinals, csma,
Nearest to gold: resorted, loft, courbet, tisza, patently, discontinued, shingles, papa,
Nearest to except: demobilized, thyroxine, vindication, journal, beamed, racer, happen, hydroxybutyrate,
Epoch 2/10 Iteration: 5100 Avg. Training loss: 4.4715 0.2332 sec/batch
Epoch 2/10 Iteration: 5200 Avg. Training loss: 4.4978 0.2281 sec/batch
Epoch 2/10 Iteration: 5300 Avg. Training loss: 4.4218 0.2305 sec/batch
Epoch 2/10 Iteration: 5400 Avg. Training loss: 4.5144 0.2281 sec/batch
Epoch 2/10 Iteration: 5500 Avg. Training loss: 4.4676 0.2290 sec/batch
Epoch 2/10 Iteration: 5600 Avg. Training loss: 4.4676 0.2274 sec/batch
Epoch 2/10 Iteration: 5700 Avg. Training loss: 4.4210 0.2289 sec/batch
Epoch 2/10 Iteration: 5800 Avg. Training loss: 4.3582 0.2276 sec/batch
Epoch 2/10 Iteration: 5900 Avg. Training loss: 4.4022 0.2285 sec/batch
Epoch 2/10 Iteration: 6000 Avg. Training loss: 4.3940 0.2293 sec/batch
Nearest to time: absorbent, jefferson, lowe, face, yage, niqqud, alf, mears,
Nearest to his: ippon, flags, prodigy, grandson, serfdom, couched, plays, davenport,
Nearest to on: hour, cleric, job, endowments, yates, lle, prut, steamy,
Nearest to many: presidente, component, maskhadov, vesicle, rexx, it, devastation, send,
Nearest to was: martha, ascertaining, bulging, hemophilia, eileen, aliyah, threatened, intermittently,
Nearest to that: isospin, arnage, scoffed, repealed, committed, musicological, distant, borromini,
Nearest to states: ocogs, children, illyrian, habitual, ambassador, faring, refined, masts,
Nearest to known: ldf, retaking, ulm, coru, breakfasts, warped, ensconced, enabling,
Nearest to active: ton, facilitated, speed, bcc, daddy, yugoslavian, nami, comprehensiveness,
Nearest to existence: assumes, nauvoo, what, aging, ostracism, pomegranates, lawgiver, emphasize,
Nearest to hold: concacaf, arbitration, specify, byng, foil, congruences, entirely, stanzas,
Nearest to pressure: wedged, normally, anfa, venerate, reactant, coated, pools, politiken,
Nearest to bible: observance, procure, everson, insights, ethnicity, canonised, monologues, mullis,
Nearest to joseph: ancient, puente, usk, carlsson, csma, paraguay, walter, brp,
Nearest to gold: resorted, courbet, loft, oda, patently, tisza, papa, kinetoscope,
Nearest to except: demobilized, thyroxine, vindication, beamed, racer, happen, journal, helles,
Epoch 2/10 Iteration: 6100 Avg. Training loss: 4.4016 0.2316 sec/batch
Epoch 2/10 Iteration: 6200 Avg. Training loss: 4.3962 0.2272 sec/batch
Epoch 2/10 Iteration: 6300 Avg. Training loss: 4.3877 0.2268 sec/batch
Epoch 2/10 Iteration: 6400 Avg. Training loss: 4.3556 0.2277 sec/batch
Epoch 2/10 Iteration: 6500 Avg. Training loss: 4.3932 0.2253 sec/batch
Epoch 2/10 Iteration: 6600 Avg. Training loss: 4.4298 0.2265 sec/batch
Epoch 2/10 Iteration: 6700 Avg. Training loss: 4.3496 0.2262 sec/batch
Epoch 2/10 Iteration: 6800 Avg. Training loss: 4.3499 0.2264 sec/batch
Epoch 2/10 Iteration: 6900 Avg. Training loss: 4.4110 0.2252 sec/batch
Epoch 2/10 Iteration: 7000 Avg. Training loss: 4.3643 0.2285 sec/batch
Nearest to time: lowe, jefferson, alf, absorbent, niqqud, face, yage, obstetrics,
Nearest to his: prodigy, ippon, flags, chani, grandson, flee, serfdom, mother,
Nearest to on: hour, cleric, yates, job, endowments, steamy, lle, prut,
Nearest to many: presidente, catharism, vesicle, maskhadov, rexx, component, send, supplant,
Nearest to was: martha, ascertaining, bulging, hemophilia, intermittently, threatened, crusading, nik,
Nearest to that: arnage, isospin, committed, scoffed, bowling, repealed, distant, musicological,
Nearest to states: illyrian, habitual, ambassador, faring, ocogs, children, vacated, masts,
Nearest to known: ldf, retaking, ulm, coru, breakfasts, frankish, voc, katherine,
Nearest to active: ton, facilitated, bcc, nami, comprehensiveness, speed, demonolators, amesbury,
Nearest to existence: nauvoo, emphasize, ostracism, what, assumes, lawgiver, pomegranates, exclude,
Nearest to hold: concacaf, arbitration, specify, entirely, congruences, stanzas, foil, rightarrow,
Nearest to pressure: wedged, reactant, normally, venerate, anfa, coated, pools, politiken,
Nearest to bible: observance, procure, insights, canonised, mullis, everson, ethnicity, tocantins,
Nearest to joseph: ancient, puente, usk, carlsson, brp, walter, paraguay, harring,
Nearest to gold: resorted, loft, courbet, oda, tisza, patently, viticulture, discontinued,
Nearest to except: demobilized, thyroxine, vindication, beamed, helles, menus, drops, racer,
Epoch 2/10 Iteration: 7100 Avg. Training loss: 4.3562 0.2328 sec/batch
Epoch 2/10 Iteration: 7200 Avg. Training loss: 4.4052 0.2291 sec/batch
Epoch 2/10 Iteration: 7300 Avg. Training loss: 4.3600 0.2287 sec/batch
Epoch 2/10 Iteration: 7400 Avg. Training loss: 4.3452 0.2280 sec/batch
Epoch 2/10 Iteration: 7500 Avg. Training loss: 4.3728 0.2271 sec/batch
Epoch 2/10 Iteration: 7600 Avg. Training loss: 4.3425 0.2275 sec/batch
Epoch 2/10 Iteration: 7700 Avg. Training loss: 4.3847 0.2275 sec/batch
Epoch 2/10 Iteration: 7800 Avg. Training loss: 4.3543 0.2269 sec/batch
Epoch 2/10 Iteration: 7900 Avg. Training loss: 4.3201 0.2312 sec/batch
Epoch 2/10 Iteration: 8000 Avg. Training loss: 4.3128 0.2285 sec/batch
Nearest to time: jefferson, niqqud, lowe, alf, absorbent, yage, cheirogaleus, domination,
Nearest to his: prodigy, flee, flags, ippon, chani, grandson, gunpowder, mother,
Nearest to on: endowments, prut, cleric, hour, job, taping, stationed, lle,
Nearest to many: presidente, devastation, supplant, vesicle, catharism, maskhadov, rexx, it,
Nearest to was: ascertaining, intermittently, crusading, threatened, hemophilia, antioch, martha, wahid,
Nearest to that: arnage, isospin, committed, scoffed, repealed, constitution, musicological, bowling,
Nearest to states: illyrian, ambassador, faring, ocogs, children, vacated, habitual, divided,
Nearest to known: ldf, retaking, ulm, frankish, ensconced, the, coru, malum,
Nearest to active: ton, facilitated, bcc, speed, comprehensiveness, nami, demonolators, amesbury,
Nearest to existence: emphasize, ostracism, nauvoo, pomegranates, lawgiver, assumes, what, pure,
Nearest to hold: arbitration, specify, concacaf, congruences, entirely, stanzas, darin, foil,
Nearest to pressure: wedged, venerate, reactant, anfa, normally, pools, internal, gases,
Nearest to bible: observance, canonised, procure, mullis, scripture, insights, everson, citations,
Nearest to joseph: ancient, puente, usk, carlsson, brp, harring, paraguay, blogger,
Nearest to gold: resorted, loft, courbet, oda, patently, tisza, viticulture, refinery,
Nearest to except: demobilized, thyroxine, beamed, vindication, amber, sandstone, helles, siberia,
Epoch 2/10 Iteration: 8100 Avg. Training loss: 4.3165 0.2322 sec/batch
Epoch 2/10 Iteration: 8200 Avg. Training loss: 4.2696 0.2291 sec/batch
Epoch 2/10 Iteration: 8300 Avg. Training loss: 4.3713 0.2290 sec/batch
Epoch 2/10 Iteration: 8400 Avg. Training loss: 4.3728 0.2281 sec/batch
Epoch 2/10 Iteration: 8500 Avg. Training loss: 4.3705 0.2273 sec/batch
Epoch 2/10 Iteration: 8600 Avg. Training loss: 4.2925 0.2278 sec/batch
Epoch 2/10 Iteration: 8700 Avg. Training loss: 4.3095 0.2256 sec/batch
Epoch 2/10 Iteration: 8800 Avg. Training loss: 4.3378 0.2292 sec/batch
Epoch 2/10 Iteration: 8900 Avg. Training loss: 4.2019 0.2293 sec/batch
Epoch 2/10 Iteration: 9000 Avg. Training loss: 4.2846 0.2299 sec/batch
Nearest to time: jefferson, niqqud, alf, domination, yage, cheirogaleus, lowe, pasture,
Nearest to his: prodigy, ippon, flags, gunpowder, lili, showman, chani, snl,
Nearest to on: prut, endowments, cleric, hour, taping, tucum, yates, steamy,
Nearest to many: devastation, presidente, rexx, messages, catharism, supplant, vesicle, timeline,
Nearest to was: ascertaining, intermittently, threatened, crusading, mandaean, martha, bourque, nik,
Nearest to that: arnage, isospin, committed, musicological, scoffed, constitution, appointment, chromaticism,
Nearest to states: illyrian, ambassador, children, ocogs, faring, vacated, priorities, outside,
Nearest to known: ldf, ulm, retaking, frankish, ensconced, malum, coru, katherine,
Nearest to active: ton, facilitated, nami, bcc, demonolators, comprehensiveness, chopper, dorrit,
Nearest to existence: nauvoo, emphasize, ostracism, pomegranates, assumes, lawgiver, document, consubstantiation,
Nearest to hold: arbitration, congruences, specify, darin, stanzas, concacaf, foil, entirely,
Nearest to pressure: wedged, venerate, anfa, reactant, normally, pools, internal, air,
Nearest to bible: observance, procure, insights, editions, canonised, mullis, commentary, citations,
Nearest to joseph: ancient, puente, usk, carlsson, colonies, paraguay, walter, brp,
Nearest to gold: resorted, loft, courbet, oda, refinery, tisza, viticulture, patently,
Nearest to except: demobilized, thyroxine, beamed, continental, vindication, amber, menus, siberia,
Epoch 2/10 Iteration: 9100 Avg. Training loss: 4.2797 0.2500 sec/batch
Epoch 2/10 Iteration: 9200 Avg. Training loss: 4.2556 0.2329 sec/batch
Epoch 3/10 Iteration: 9300 Avg. Training loss: 4.3387 0.1103 sec/batch
Epoch 3/10 Iteration: 9400 Avg. Training loss: 4.2121 0.2303 sec/batch
Epoch 3/10 Iteration: 9500 Avg. Training loss: 4.1754 0.2306 sec/batch
Epoch 3/10 Iteration: 9600 Avg. Training loss: 4.1810 0.2321 sec/batch
Epoch 3/10 Iteration: 9700 Avg. Training loss: 4.1870 0.2309 sec/batch
Epoch 3/10 Iteration: 9800 Avg. Training loss: 4.1702 0.2321 sec/batch
Epoch 3/10 Iteration: 9900 Avg. Training loss: 4.1871 0.2325 sec/batch
Epoch 3/10 Iteration: 10000 Avg. Training loss: 4.1459 0.2317 sec/batch
Nearest to time: jefferson, niqqud, yage, lowe, domination, alf, cheirogaleus, pasture,
Nearest to his: gunpowder, he, flags, prodigy, ippon, flee, chani, snl,
Nearest to on: prut, cleric, hour, endowments, apollo, lle, steamy, taping,
Nearest to many: catharism, devastation, presidente, supplant, vesicle, maskhadov, ensues, rexx,
Nearest to was: ascertaining, crusading, intermittently, threatened, bfbs, martha, bourque, falconer,
Nearest to that: arnage, isospin, musicological, committed, scoffed, constitution, chromaticism, distant,
Nearest to states: ambassador, illyrian, children, faring, ocogs, priorities, vacated, united,
Nearest to known: ldf, ulm, retaking, frankish, salutis, ensconced, malum, name,
Nearest to active: ton, facilitated, demonolators, nami, bcc, comprehensiveness, amesbury, dorrit,
Nearest to existence: nauvoo, ostracism, assumes, emphasize, pure, consubstantiation, lawgiver, pomegranates,
Nearest to hold: arbitration, specify, congruences, stanzas, concacaf, unheard, foil, entirely,
Nearest to pressure: wedged, normally, reactant, anfa, venerate, pools, temperature, gases,
Nearest to bible: editions, observance, procure, canonised, scripture, citations, everson, commentary,
Nearest to joseph: ancient, puente, usk, carlsson, mendelssohn, harring, brp, colonies,
Nearest to gold: loft, resorted, courbet, refinery, oda, reactivity, tisza, viticulture,
Nearest to except: demobilized, thyroxine, beamed, continental, sandstone, amber, menus, drops,
Epoch 3/10 Iteration: 10100 Avg. Training loss: 4.2338 0.2338 sec/batch
Epoch 3/10 Iteration: 10200 Avg. Training loss: 4.2053 0.2409 sec/batch
Epoch 3/10 Iteration: 10300 Avg. Training loss: 4.2029 0.2312 sec/batch
Epoch 3/10 Iteration: 10400 Avg. Training loss: 4.0798 0.2316 sec/batch
Epoch 3/10 Iteration: 10500 Avg. Training loss: 4.1509 0.2356 sec/batch
Epoch 3/10 Iteration: 10600 Avg. Training loss: 4.1312 0.2318 sec/batch
Epoch 3/10 Iteration: 10700 Avg. Training loss: 4.1301 0.2319 sec/batch
Epoch 3/10 Iteration: 10800 Avg. Training loss: 4.1650 0.2328 sec/batch
Epoch 3/10 Iteration: 10900 Avg. Training loss: 4.1627 0.2320 sec/batch
Epoch 3/10 Iteration: 11000 Avg. Training loss: 4.1210 0.2339 sec/batch
Nearest to time: jefferson, niqqud, lowe, yage, builder, alf, domination, sanitization,
Nearest to his: he, prodigy, ippon, gunpowder, flags, chani, snl, paternal,
Nearest to on: prut, cleric, endowments, lle, hour, markings, taping, freedb,
Nearest to many: catharism, supplant, presidente, timeline, maskhadov, devastation, rexx, ensues,
Nearest to was: ascertaining, threatened, bourque, intermittently, crusading, bfbs, martha, huevos,
Nearest to that: isospin, committed, arnage, musicological, chromaticism, bowling, distant, scoffed,
Nearest to states: illyrian, ambassador, united, children, faring, priorities, ocogs, sadistic,
Nearest to known: ulm, retaking, ldf, malum, trapani, salutis, frankish, ensconced,
Nearest to active: ton, facilitated, demonolators, nami, comprehensiveness, bcc, yugoslavian, amesbury,
Nearest to existence: nauvoo, ostracism, emphasize, assumes, lawgiver, pure, consubstantiation, schelter,
Nearest to hold: specify, arbitration, stanzas, congruences, foil, petition, concacaf, entirely,
Nearest to pressure: reactant, wedged, normally, venerate, anfa, pools, overtone, gases,
Nearest to bible: editions, commentary, observance, canonised, procure, scripture, citations, mullis,
Nearest to joseph: ancient, puente, usk, harring, carlsson, colonies, mendelssohn, fenwick,
Nearest to gold: loft, resorted, courbet, oda, refinery, viticulture, reactivity, crystallize,
Nearest to except: demobilized, thyroxine, beamed, helles, drops, continental, vindication, amber,
Epoch 3/10 Iteration: 11100 Avg. Training loss: 4.1375 0.2334 sec/batch
Epoch 3/10 Iteration: 11200 Avg. Training loss: 4.1653 0.2313 sec/batch
Epoch 3/10 Iteration: 11300 Avg. Training loss: 4.1507 0.2303 sec/batch
Epoch 3/10 Iteration: 11400 Avg. Training loss: 4.1199 0.2297 sec/batch
Epoch 3/10 Iteration: 11500 Avg. Training loss: 4.1457 0.2304 sec/batch
Epoch 3/10 Iteration: 11600 Avg. Training loss: 4.1511 0.2289 sec/batch
Epoch 3/10 Iteration: 11700 Avg. Training loss: 4.1754 0.2319 sec/batch
Epoch 3/10 Iteration: 11800 Avg. Training loss: 4.1186 0.2316 sec/batch
Epoch 3/10 Iteration: 11900 Avg. Training loss: 4.1096 0.2333 sec/batch
Epoch 3/10 Iteration: 12000 Avg. Training loss: 4.1589 0.2317 sec/batch
Nearest to time: jefferson, niqqud, alf, lowe, finish, builder, obstetrics, sanitization,
Nearest to his: he, prodigy, partisanship, flags, gunpowder, chani, consulted, snl,
Nearest to on: taping, prut, endowments, freedb, the, markings, hour, katsuhiro,
Nearest to many: supplant, catharism, timeline, presidente, ensues, devastation, rexx, genetics,
Nearest to was: ascertaining, intermittently, threatened, preserver, antioch, crusading, bourque, barnabas,
Nearest to that: committed, arnage, isospin, musicological, bowling, distant, chromaticism, constitution,
Nearest to states: united, ambassador, illyrian, priorities, faring, sadistic, presidential, unrepresented,
Nearest to known: ulm, retaking, ldf, frankish, ensconced, name, salutis, malum,
Nearest to active: ton, facilitated, bcc, nami, demonolators, yugoslavian, chopper, comprehensiveness,
Nearest to existence: nauvoo, ostracism, assumes, pure, emphasize, lawgiver, what, urpmi,
Nearest to hold: specify, stanzas, arbitration, rightarrow, foil, congruences, concacaf, entirely,
Nearest to pressure: wedged, reactant, normally, pools, venerate, rest, salute, anfa,
Nearest to bible: editions, commentary, observance, citations, procure, mullis, scripture, everson,
Nearest to joseph: ancient, puente, usk, carlsson, colonies, fenwick, augustinians, brp,
Nearest to gold: loft, oda, courbet, resorted, refinery, reactivity, viticulture, crystallize,
Nearest to except: demobilized, beamed, vindication, drops, helles, thyroxine, continental, amber,
Epoch 3/10 Iteration: 12100 Avg. Training loss: 4.1723 0.2334 sec/batch
Epoch 3/10 Iteration: 12200 Avg. Training loss: 4.1584 0.2321 sec/batch
Epoch 3/10 Iteration: 12300 Avg. Training loss: 4.1555 0.2303 sec/batch
Epoch 3/10 Iteration: 12400 Avg. Training loss: 4.1640 0.2307 sec/batch
Epoch 3/10 Iteration: 12500 Avg. Training loss: 4.1247 0.2302 sec/batch
Epoch 3/10 Iteration: 12600 Avg. Training loss: 4.0628 0.2306 sec/batch
Epoch 3/10 Iteration: 12700 Avg. Training loss: 4.1432 0.2326 sec/batch
Epoch 3/10 Iteration: 12800 Avg. Training loss: 4.0951 0.2336 sec/batch
Epoch 3/10 Iteration: 12900 Avg. Training loss: 4.1799 0.2325 sec/batch
Epoch 3/10 Iteration: 13000 Avg. Training loss: 4.1967 0.2303 sec/batch
Nearest to time: jefferson, niqqud, alf, builder, pattie, domination, sanitization, lowe,
Nearest to his: he, prodigy, flags, gus, partisanship, snl, obe, showman,
Nearest to on: prut, the, endowments, cleric, tuc, lle, bana, apollo,
Nearest to many: supplant, catharism, presidente, swooping, ensues, devastation, proscription, timeline,
Nearest to was: intermittently, ascertaining, preserver, the, threatened, antioch, crusading, huevos,
Nearest to that: committed, arnage, musicological, constitution, isospin, distant, chromaticism, bowling,
Nearest to states: united, ambassador, children, isonzo, runnymede, illyrian, menzies, vacated,
Nearest to known: ulm, retaking, ldf, frankish, moabite, ensconced, trapani, malum,
Nearest to active: facilitated, ton, nami, yugoslavian, bcc, amesbury, dorrit, demonolators,
Nearest to existence: nauvoo, ostracism, pure, urpmi, assumes, consubstantiation, emphasize, schelter,
Nearest to hold: arbitration, specify, darin, stanzas, concacaf, petition, penitential, unspecified,
Nearest to pressure: wedged, reactant, normally, fervently, salute, internal, pools, anfa,
Nearest to bible: editions, observance, commentary, citations, astrologer, canonised, mullis, scripture,
Nearest to joseph: ancient, puente, usk, colonies, paraguay, fenwick, carlsson, brp,
Nearest to gold: loft, oda, courbet, refinery, resorted, viticulture, crystallize, kinetoscope,
Nearest to except: demobilized, beamed, drops, helles, amber, vindication, continental, thyroxine,
Epoch 3/10 Iteration: 13100 Avg. Training loss: 4.2113 0.2355 sec/batch
Epoch 3/10 Iteration: 13200 Avg. Training loss: 4.0875 0.2301 sec/batch
Epoch 3/10 Iteration: 13300 Avg. Training loss: 4.1097 0.2283 sec/batch
Epoch 3/10 Iteration: 13400 Avg. Training loss: 4.1278 0.2293 sec/batch
Epoch 3/10 Iteration: 13500 Avg. Training loss: 4.0352 0.2306 sec/batch
Epoch 3/10 Iteration: 13600 Avg. Training loss: 4.1533 0.2385 sec/batch
Epoch 3/10 Iteration: 13700 Avg. Training loss: 4.1363 0.2723 sec/batch
Epoch 3/10 Iteration: 13800 Avg. Training loss: 4.1049 0.2522 sec/batch
Epoch 4/10 Iteration: 13900 Avg. Training loss: 4.1543 0.0559 sec/batch
Epoch 4/10 Iteration: 14000 Avg. Training loss: 4.0760 0.2427 sec/batch
Nearest to time: jefferson, niqqud, sanitization, builder, domination, obstetrics, alf, pattie,
Nearest to his: he, snl, choking, prodigy, chani, gus, consulted, flags,
Nearest to on: prut, the, endowments, taping, apollo, hour, katsuhiro, bana,
Nearest to many: supplant, catharism, genetics, swooping, jit, messages, devastation, timeline,
Nearest to was: intermittently, ascertaining, threatened, preserver, crusading, emails, ordnance, huevos,
Nearest to that: arnage, committed, isospin, musicological, chromaticism, constitution, distant, axiomatization,
Nearest to states: united, ambassador, faring, isonzo, ocogs, children, sadistic, sula,
Nearest to known: ulm, ldf, retaking, frankish, moabite, salutis, malum, demophon,
Nearest to active: facilitated, ton, nami, yugoslavian, comprehensiveness, amesbury, dorrit, demonolators,
Nearest to existence: nauvoo, ostracism, pure, assumes, emphasize, quixtar, consubstantiation, urpmi,
Nearest to hold: specify, arbitration, stanzas, concacaf, darin, congruences, unspecified, foil,
Nearest to pressure: reactant, normally, wedged, pools, internal, overtone, fervently, salute,
Nearest to bible: editions, observance, astrologer, commentary, canonised, scripture, citations, procure,
Nearest to joseph: ancient, puente, colonies, usk, brp, paraguay, mendelssohn, fenwick,
Nearest to gold: loft, oda, courbet, refinery, resorted, crystallize, sacred, kinetoscope,
Nearest to except: demobilized, beamed, drops, thyroxine, vr, helles, amber, hydroxybutyrate,
Epoch 4/10 Iteration: 14100 Avg. Training loss: 4.0177 0.2450 sec/batch
Epoch 4/10 Iteration: 14200 Avg. Training loss: 4.0177 0.2485 sec/batch
Epoch 4/10 Iteration: 14300 Avg. Training loss: 4.0648 0.2483 sec/batch
Epoch 4/10 Iteration: 14400 Avg. Training loss: 3.9919 0.2443 sec/batch
Epoch 4/10 Iteration: 14500 Avg. Training loss: 4.0534 0.2455 sec/batch
Epoch 4/10 Iteration: 14600 Avg. Training loss: 4.0090 0.2430 sec/batch
Epoch 4/10 Iteration: 14700 Avg. Training loss: 4.0461 0.2425 sec/batch
Epoch 4/10 Iteration: 14800 Avg. Training loss: 4.0258 0.2541 sec/batch
Epoch 4/10 Iteration: 14900 Avg. Training loss: 4.0962 0.2562 sec/batch
Epoch 4/10 Iteration: 15000 Avg. Training loss: 3.9836 0.2586 sec/batch
Nearest to time: jefferson, finish, niqqud, builder, spss, alf, lowe, sanitization,
Nearest to his: he, prodigy, snl, obe, choking, jealous, capet, flags,
Nearest to on: the, prut, apollo, markings, endowments, lle, hour, taping,
Nearest to many: supplant, genetics, catharism, swooping, jit, ensues, maskhadov, proscription,
Nearest to was: the, intermittently, had, ascertaining, threatened, bourque, huevos, antioch,
Nearest to that: committed, arnage, isospin, musicological, chromaticism, constitution, distant, bowling,
Nearest to states: united, faring, ocogs, ambassador, sula, illyrian, batoche, isonzo,
Nearest to known: ulm, name, ldf, retaking, frankish, malum, oklahoma, the,
Nearest to active: facilitated, ton, nami, yugoslavian, comprehensiveness, bcc, sippar, dorrit,
Nearest to existence: nauvoo, ostracism, emphasize, assumes, consubstantiation, pure, uninformed, teleological,
Nearest to hold: specify, arbitration, concacaf, stanzas, darin, gift, foil, rightarrow,
Nearest to pressure: reactant, normally, overtone, wedged, internal, fervently, gases, impartiality,
Nearest to bible: editions, observance, commentary, mamre, scripture, astrologer, citations, procure,
Nearest to joseph: ancient, puente, usk, colonies, harring, fenwick, peart, successions,
Nearest to gold: loft, oda, courbet, refinery, resorted, viticulture, reactivity, tisza,
Nearest to except: demobilized, beamed, drops, vr, sandstone, helles, vindication, amber,
Epoch 4/10 Iteration: 15100 Avg. Training loss: 3.9690 0.2618 sec/batch
Epoch 4/10 Iteration: 15200 Avg. Training loss: 4.0308 0.2486 sec/batch
Epoch 4/10 Iteration: 15300 Avg. Training loss: 3.9997 0.2432 sec/batch
Epoch 4/10 Iteration: 15400 Avg. Training loss: 4.0036 0.2438 sec/batch
Epoch 4/10 Iteration: 15500 Avg. Training loss: 4.0744 0.2437 sec/batch
Epoch 4/10 Iteration: 15600 Avg. Training loss: 3.9950 0.2480 sec/batch
Epoch 4/10 Iteration: 15700 Avg. Training loss: 4.0060 0.2439 sec/batch
Epoch 4/10 Iteration: 15800 Avg. Training loss: 4.0761 0.2436 sec/batch
Epoch 4/10 Iteration: 15900 Avg. Training loss: 4.0074 0.2454 sec/batch
Epoch 4/10 Iteration: 16000 Avg. Training loss: 4.0149 0.2449 sec/batch
Nearest to time: builder, finish, niqqud, lowe, spss, jefferson, alf, obstetrics,
Nearest to his: he, prodigy, lili, snl, ippon, obe, jealous, paternal,
Nearest to on: the, endowments, prut, markings, katsuhiro, fragmenting, raeben, greg,
Nearest to many: genetics, supplant, catharism, jit, swooping, messages, colouring, candidiasis,
Nearest to was: ascertaining, intermittently, the, threatened, huevos, crusading, antioch, jurats,
Nearest to that: committed, arnage, isospin, musicological, chromaticism, is, constitution, distant,
Nearest to states: united, ambassador, unrepresented, faring, vacated, sadistic, ocogs, sula,
Nearest to known: ulm, name, salutis, ldf, malum, retaking, ensconced, oklahoma,
Nearest to active: facilitated, ton, nami, comprehensiveness, yugoslavian, bcc, dorrit, sippar,
Nearest to existence: nauvoo, ostracism, assumes, pure, emphasize, schelter, consubstantiation, offsets,
Nearest to hold: specify, arbitration, concacaf, darin, eminem, penitential, gift, stanzas,
Nearest to pressure: reactant, normally, overtone, temperature, wedged, impartiality, internal, atpase,
Nearest to bible: editions, observance, commentary, mamre, citations, astrologer, canonised, serenity,
Nearest to joseph: ancient, puente, usk, colonies, peart, fenwick, harring, paraguay,
Nearest to gold: loft, oda, refinery, courbet, viticulture, resorted, reactivity, atyrau,
Nearest to except: demobilized, beamed, vr, drops, helles, vindication, hydroxybutyrate, sandstone,
Epoch 4/10 Iteration: 16100 Avg. Training loss: 4.0219 0.2458 sec/batch
Epoch 4/10 Iteration: 16200 Avg. Training loss: 4.0449 0.2503 sec/batch
Epoch 4/10 Iteration: 16300 Avg. Training loss: 4.0652 0.2532 sec/batch
Epoch 4/10 Iteration: 16400 Avg. Training loss: 4.0013 0.2505 sec/batch
Epoch 4/10 Iteration: 16500 Avg. Training loss: 4.0461 0.2471 sec/batch
Epoch 4/10 Iteration: 16600 Avg. Training loss: 4.0383 0.2477 sec/batch
Epoch 4/10 Iteration: 16700 Avg. Training loss: 4.0361 0.2485 sec/batch
Epoch 4/10 Iteration: 16800 Avg. Training loss: 4.0214 0.2483 sec/batch
Epoch 4/10 Iteration: 16900 Avg. Training loss: 4.0280 0.2463 sec/batch
Epoch 4/10 Iteration: 17000 Avg. Training loss: 4.0238 0.2469 sec/batch
Nearest to time: finish, builder, domination, alf, jefferson, niqqud, sanitization, because,
Nearest to his: he, prodigy, snl, obe, paternal, to, partisanship, remarkable,
Nearest to on: the, prut, endowments, fragmenting, epimetheus, katsuhiro, freedb, nineveh,
Nearest to many: supplant, genetics, catharism, swooping, candidiasis, ensues, jit, treasure,
Nearest to was: the, intermittently, had, ascertaining, of, antioch, huevos, crusading,
Nearest to that: committed, arnage, musicological, constitution, distant, bowling, chromaticism, isospin,
Nearest to states: united, ambassador, ocogs, isonzo, vacated, batoche, act, sula,
Nearest to known: ulm, frankish, ldf, salutis, retaking, oklahoma, name, trapani,
Nearest to active: facilitated, ton, nami, yugoslavian, comprehensiveness, bcc, dorrit, sippar,
Nearest to existence: ostracism, nauvoo, urpmi, assumes, consubstantiation, schelter, pure, uninformed,
Nearest to hold: specify, arbitration, penitential, gift, eminem, darin, unspecified, respect,
Nearest to pressure: reactant, fervently, salute, internal, normally, pools, temperature, overtone,
Nearest to bible: observance, editions, commentary, citations, canonised, mamre, norsemen, scripture,
Nearest to joseph: ancient, puente, usk, colonies, paraguay, fenwick, harring, peart,
Nearest to gold: loft, oda, refinery, viticulture, courbet, reactivity, resorted, figs,
Nearest to except: demobilized, beamed, vr, drops, helles, vindication, reassured, happen,
Epoch 4/10 Iteration: 17100 Avg. Training loss: 4.0428 0.2512 sec/batch
Epoch 4/10 Iteration: 17200 Avg. Training loss: 3.9936 0.2454 sec/batch
Epoch 4/10 Iteration: 17300 Avg. Training loss: 3.9747 0.2464 sec/batch
Epoch 4/10 Iteration: 17400 Avg. Training loss: 4.0423 0.2498 sec/batch
Epoch 4/10 Iteration: 17500 Avg. Training loss: 4.0802 0.2499 sec/batch
Epoch 4/10 Iteration: 17600 Avg. Training loss: 4.0921 0.2526 sec/batch
Epoch 4/10 Iteration: 17700 Avg. Training loss: 4.1055 0.2472 sec/batch
Epoch 4/10 Iteration: 17800 Avg. Training loss: 4.0233 0.2480 sec/batch
Epoch 4/10 Iteration: 17900 Avg. Training loss: 3.9817 0.2466 sec/batch
Epoch 4/10 Iteration: 18000 Avg. Training loss: 4.0122 0.2500 sec/batch
Nearest to time: builder, finish, sanitization, alf, niqqud, domination, jefferson, because,
Nearest to his: he, prodigy, lili, obe, paternal, wolfe, showman, snl,
Nearest to on: the, prut, endowments, revitalised, woodhouse, fragmenting, katsuhiro, trilateral,
Nearest to many: swooping, genetics, catharism, supplant, ensues, jit, candidiasis, pears,
Nearest to was: the, had, huevos, threatened, ascertaining, intermittently, crusading, antioch,
Nearest to that: committed, arnage, musicological, distant, isospin, chromaticism, constitution, bowling,
Nearest to states: united, ambassador, ocogs, gpo, sula, act, isonzo, america,
Nearest to known: ulm, oklahoma, demophon, tonga, frankish, ldf, retaking, salutis,
Nearest to active: facilitated, nami, ton, yugoslavian, comprehensiveness, bcc, sippar, dorrit,
Nearest to existence: ostracism, nauvoo, pacifists, schelter, urpmi, consubstantiation, subordinated, uninformed,
Nearest to hold: specify, arbitration, penitential, eminem, darin, respect, gift, unspecified,
Nearest to pressure: reactant, internal, fervently, salute, pools, normally, ndel, impartiality,
Nearest to bible: editions, observance, citations, commentary, canonised, mamre, astrologer, mullis,
Nearest to joseph: ancient, puente, usk, colonies, paraguay, fenwick, harring, carlsson,
Nearest to gold: loft, oda, refinery, courbet, viticulture, figs, atyrau, reactivity,
Nearest to except: demobilized, beamed, vr, drops, helles, hydroxybutyrate, vindication, happen,
Epoch 4/10 Iteration: 18100 Avg. Training loss: 3.9522 0.2489 sec/batch
Epoch 4/10 Iteration: 18200 Avg. Training loss: 4.0186 0.2472 sec/batch
Epoch 4/10 Iteration: 18300 Avg. Training loss: 4.0215 0.2455 sec/batch
Epoch 4/10 Iteration: 18400 Avg. Training loss: 4.0318 0.2465 sec/batch
Epoch 4/10 Iteration: 18500 Avg. Training loss: 4.0569 0.2493 sec/batch
Epoch 5/10 Iteration: 18600 Avg. Training loss: 3.9750 0.2388 sec/batch
Epoch 5/10 Iteration: 18700 Avg. Training loss: 3.9725 0.2467 sec/batch
Epoch 5/10 Iteration: 18800 Avg. Training loss: 3.9279 0.2493 sec/batch
Epoch 5/10 Iteration: 18900 Avg. Training loss: 3.9800 0.2562 sec/batch
Epoch 5/10 Iteration: 19000 Avg. Training loss: 3.9180 0.2464 sec/batch
Nearest to time: finish, niqqud, builder, alf, domination, sanitization, absorbent, spss,
Nearest to his: he, lili, degas, bestowed, jealous, theses, prodigy, son,
Nearest to on: the, apollo, revitalised, prut, markings, raeben, thro, endowments,
Nearest to many: swooping, genetics, catharism, supplant, jit, treasure, ensues, attacker,
Nearest to was: the, had, ascertaining, huevos, reconstructed, intermittently, as, spent,
Nearest to that: committed, arnage, musicological, isospin, to, chromaticism, distant, constitution,
Nearest to states: united, ocogs, faring, tennessee, gpo, sula, america, ambassador,
Nearest to known: ulm, demophon, salutis, oklahoma, name, tonga, malum, frankish,
Nearest to active: facilitated, nami, yugoslavian, ton, dorrit, bcc, comprehensiveness, sippar,
Nearest to existence: nauvoo, ostracism, pacifists, consubstantiation, subordinated, schelter, uninformed, pure,
Nearest to hold: specify, arbitration, gift, darin, hypothesis, unspecified, respect, penitential,
Nearest to pressure: reactant, internal, normally, pools, overtone, atpase, temperature, ndel,
Nearest to bible: editions, observance, citations, commentary, canonised, norsemen, therapeutae, sifra,
Nearest to joseph: ancient, puente, usk, harring, colonies, peart, paraguay, fenwick,
Nearest to gold: loft, oda, refinery, silver, atyrau, courbet, figs, resorted,
Nearest to except: vr, demobilized, beamed, drops, helles, hydroxybutyrate, match, sandstone,
Epoch 5/10 Iteration: 19100 Avg. Training loss: 3.9436 0.2505 sec/batch
Epoch 5/10 Iteration: 19200 Avg. Training loss: 3.9019 0.2477 sec/batch
Epoch 5/10 Iteration: 19300 Avg. Training loss: 3.9766 0.2447 sec/batch
Epoch 5/10 Iteration: 19400 Avg. Training loss: 3.9733 0.2459 sec/batch
Epoch 5/10 Iteration: 19500 Avg. Training loss: 3.9781 0.2600 sec/batch
Epoch 5/10 Iteration: 19600 Avg. Training loss: 3.9714 0.2495 sec/batch
Epoch 5/10 Iteration: 19700 Avg. Training loss: 3.8591 0.2589 sec/batch
Epoch 5/10 Iteration: 19800 Avg. Training loss: 3.9478 0.2563 sec/batch
Epoch 5/10 Iteration: 19900 Avg. Training loss: 3.8914 0.2618 sec/batch
Epoch 5/10 Iteration: 20000 Avg. Training loss: 3.9644 0.2574 sec/batch
Nearest to time: finish, sanitization, niqqud, spss, builder, domination, alf, absorbent,
Nearest to his: he, prodigy, obe, lili, jealous, theses, unsuited, degas,
Nearest to on: the, revitalised, endowments, markings, prut, fragmenting, evapotranspiration, epimetheus,
Nearest to many: genetics, swooping, jit, supplant, catharism, attacker, ensues, it,
Nearest to was: the, had, reconstructed, huevos, threatened, ascertaining, spent, intermittently,
Nearest to that: committed, to, musicological, distant, is, arnage, isospin, the,
Nearest to states: united, america, ocogs, faring, of, tennessee, sula, gpo,
Nearest to known: ulm, demophon, name, oklahoma, malum, salutis, tonga, moabite,
Nearest to active: facilitated, yugoslavian, nami, ton, comprehensiveness, sippar, bcc, dissolves,
Nearest to existence: nauvoo, ostracism, pacifists, subordinated, knapsack, urpmi, schelter, azrael,
Nearest to hold: specify, gift, darin, arbitration, unspecified, eminem, rightarrow, hypothesis,
Nearest to pressure: reactant, pools, temperature, normally, overtone, impartiality, newton, ndel,
Nearest to bible: editions, observance, commentary, canonised, citations, sifra, therapeutae, akiva,
Nearest to joseph: ancient, puente, usk, fenwick, harring, thyself, colonies, peart,
Nearest to gold: loft, oda, silver, refinery, figs, courbet, leonis, atyrau,
Nearest to except: vr, demobilized, beamed, drops, match, hydroxybutyrate, racer, sandstone,
Epoch 5/10 Iteration: 20100 Avg. Training loss: 3.9476 0.2535 sec/batch
Epoch 5/10 Iteration: 20200 Avg. Training loss: 3.9247 0.2512 sec/batch
Epoch 5/10 Iteration: 20300 Avg. Training loss: 3.9371 0.2495 sec/batch
Epoch 5/10 Iteration: 20400 Avg. Training loss: 3.9567 0.2507 sec/batch
Epoch 5/10 Iteration: 20500 Avg. Training loss: 3.9902 0.2517 sec/batch
Epoch 5/10 Iteration: 20600 Avg. Training loss: 3.9288 0.2509 sec/batch
Epoch 5/10 Iteration: 20700 Avg. Training loss: 3.9401 0.2570 sec/batch
Epoch 5/10 Iteration: 20800 Avg. Training loss: 3.9549 0.2562 sec/batch
Epoch 5/10 Iteration: 20900 Avg. Training loss: 3.9681 0.2584 sec/batch
Epoch 5/10 Iteration: 21000 Avg. Training loss: 3.9571 0.2538 sec/batch
Nearest to time: finish, niqqud, alf, sanitization, builder, spss, domination, lowe,
Nearest to his: he, prodigy, bestowed, obe, anxious, degas, theses, jealous,
Nearest to on: the, katsuhiro, evapotranspiration, fragmenting, revitalised, thirteen, trilateral, epimetheus,
Nearest to many: genetics, swooping, candidiasis, catharism, treasure, ensues, supplant, attacker,
Nearest to was: the, had, ascertaining, reconstructed, huevos, as, abrogate, preserver,
Nearest to that: committed, to, distant, arnage, musicological, is, canonically, midland,
Nearest to states: united, america, faring, ocogs, tennessee, isonzo, of, liechtenstein,
Nearest to known: name, salutis, ulm, malum, oklahoma, demophon, moabite, ensconced,
Nearest to active: facilitated, nami, yugoslavian, ton, comprehensiveness, dorrit, bcc, sippar,
Nearest to existence: nauvoo, ostracism, subordinated, cyclotron, urpmi, pacifists, schelter, mandible,
Nearest to hold: specify, darin, eminem, bullpen, gift, rightarrow, unspecified, celebrate,
Nearest to pressure: reactant, impartiality, fervently, salute, pools, temperature, ndel, internal,
Nearest to bible: editions, observance, citations, canonised, commentary, therapeutae, serenity, akiva,
Nearest to joseph: ancient, puente, fenwick, usk, harring, paraguay, thyself, peart,
Nearest to gold: loft, oda, refinery, silver, figs, atyrau, leonis, viticulture,
Nearest to except: vr, demobilized, beamed, match, drops, racer, helles, consecrated,
Epoch 5/10 Iteration: 21100 Avg. Training loss: 3.9649 0.2575 sec/batch
Epoch 5/10 Iteration: 21200 Avg. Training loss: 3.9322 0.2521 sec/batch
Epoch 5/10 Iteration: 21300 Avg. Training loss: 3.9326 0.2519 sec/batch
Epoch 5/10 Iteration: 21400 Avg. Training loss: 3.9459 0.2524 sec/batch
Epoch 5/10 Iteration: 21500 Avg. Training loss: 3.9876 0.2501 sec/batch
Epoch 5/10 Iteration: 21600 Avg. Training loss: 3.9625 0.2535 sec/batch
Epoch 5/10 Iteration: 21700 Avg. Training loss: 3.9599 0.2389 sec/batch
Epoch 5/10 Iteration: 21800 Avg. Training loss: 3.9402 0.2376 sec/batch
Epoch 5/10 Iteration: 21900 Avg. Training loss: 3.9062 0.2410 sec/batch
Epoch 5/10 Iteration: 22000 Avg. Training loss: 3.9800 0.2424 sec/batch
Nearest to time: domination, niqqud, finish, sanitization, alf, terminates, spss, builder,
Nearest to his: he, prodigy, obe, governorship, degas, anxious, calvi, unsuited,
Nearest to on: the, trilateral, prut, thirteen, endowments, shillings, fragmenting, falconer,
Nearest to many: swooping, genetics, supplant, candidiasis, ensues, catharism, dss, are,
Nearest to was: the, had, ascertaining, reconstructed, huevos, pershing, as, of,
Nearest to that: committed, to, musicological, distant, arnage, canonically, midland, is,
Nearest to states: united, of, america, ocogs, faring, tennessee, ambassador, batoche,
Nearest to known: malum, moabite, salutis, name, frankish, ulm, the, oklahoma,
Nearest to active: facilitated, nami, yugoslavian, ton, dissolves, comprehensiveness, saavedra, heterosexuals,
Nearest to existence: nauvoo, ostracism, mandible, cyclotron, subordinated, begets, azrael, urpmi,
Nearest to hold: darin, specify, eminem, gift, celebrate, respect, arbitration, unspecified,
Nearest to pressure: fervently, reactant, impartiality, salute, internal, pools, ndel, air,
Nearest to bible: editions, observance, citations, canonised, therapeutae, akiva, serenity, norsemen,
Nearest to joseph: ancient, puente, usk, fenwick, paraguay, harring, waitakere, thyself,
Nearest to gold: oda, loft, silver, refinery, viticulture, atyrau, figs, courbet,
Nearest to except: vr, beamed, demobilized, drops, consecrated, match, continental, sandstone,
Epoch 5/10 Iteration: 22100 Avg. Training loss: 3.9292 0.2478 sec/batch
Epoch 5/10 Iteration: 22200 Avg. Training loss: 4.0643 0.2412 sec/batch
Epoch 5/10 Iteration: 22300 Avg. Training loss: 4.0029 0.2395 sec/batch
Epoch 5/10 Iteration: 22400 Avg. Training loss: 3.9821 0.2375 sec/batch
Epoch 5/10 Iteration: 22500 Avg. Training loss: 3.9065 0.2384 sec/batch
Epoch 5/10 Iteration: 22600 Avg. Training loss: 3.8795 0.2363 sec/batch
Epoch 5/10 Iteration: 22700 Avg. Training loss: 3.9659 0.2361 sec/batch
Epoch 5/10 Iteration: 22800 Avg. Training loss: 3.8882 0.2382 sec/batch
Epoch 5/10 Iteration: 22900 Avg. Training loss: 3.9246 0.2418 sec/batch
Epoch 5/10 Iteration: 23000 Avg. Training loss: 3.9385 0.2363 sec/batch
Nearest to time: domination, sanitization, niqqud, builder, pattie, finish, eight, hymnal,
Nearest to his: he, degas, unsuited, prodigy, obe, calvi, showman, mccartney,
Nearest to on: the, evapotranspiration, trilateral, thirteen, prut, revitalised, apollo, same,
Nearest to many: swooping, genetics, candidiasis, treasure, jit, are, colouring, used,
Nearest to was: the, had, as, ascertaining, reconstructed, nls, time, huevos,
Nearest to that: committed, to, distant, arnage, hallucinatory, musicological, canonically, the,
Nearest to states: united, america, tennessee, ocogs, faring, of, batoche, gpo,
Nearest to known: malum, moabite, oklahoma, webzine, ulm, salutis, demophon, name,
Nearest to active: facilitated, nami, yugoslavian, saavedra, dissolves, ton, comprehensiveness, heterosexuals,
Nearest to existence: nauvoo, ostracism, subordinated, topkapi, pacifists, mandible, azrael, begets,
Nearest to hold: darin, unspecified, eminem, specify, hypothesis, arbitration, gift, bullpen,
Nearest to pressure: reactant, fervently, internal, temperature, air, impartiality, salute, pools,
Nearest to bible: editions, observance, citations, canonised, norsemen, tractates, therapeutae, akiva,
Nearest to joseph: ancient, puente, usk, fenwick, paraguay, harring, waitakere, thyself,
Nearest to gold: oda, loft, silver, refinery, figs, viticulture, leonis, courbet,
Nearest to except: vr, demobilized, beamed, match, drops, hydroxybutyrate, kushner, conserved,
Epoch 5/10 Iteration: 23100 Avg. Training loss: 3.9507 0.2399 sec/batch
Epoch 6/10 Iteration: 23200 Avg. Training loss: 3.9544 0.1658 sec/batch
Epoch 6/10 Iteration: 23300 Avg. Training loss: 3.9140 0.2383 sec/batch
Epoch 6/10 Iteration: 23400 Avg. Training loss: 3.8719 0.2381 sec/batch
Epoch 6/10 Iteration: 23500 Avg. Training loss: 3.9331 0.2366 sec/batch
Epoch 6/10 Iteration: 23600 Avg. Training loss: 3.8955 0.2381 sec/batch
Epoch 6/10 Iteration: 23700 Avg. Training loss: 3.9097 0.2426 sec/batch
Epoch 6/10 Iteration: 23800 Avg. Training loss: 3.8713 0.2413 sec/batch
Epoch 6/10 Iteration: 23900 Avg. Training loss: 3.9211 0.2377 sec/batch
Epoch 6/10 Iteration: 24000 Avg. Training loss: 3.9255 0.2355 sec/batch
Nearest to time: finish, was, domination, eight, niqqud, builder, hymnal, pattie,
Nearest to his: he, prodigy, unsuited, to, had, obe, governorship, snl,
Nearest to on: the, apollo, thirteen, markings, was, revitalised, mcneil, steamy,
Nearest to many: swooping, treasure, candidiasis, genetics, used, ensues, known, are,
Nearest to was: the, had, time, as, but, of, success, spent,
Nearest to that: committed, distant, to, the, is, arnage, canonically, chromaticism,
Nearest to states: united, of, america, faring, ocogs, univision, tennessee, batoche,
Nearest to known: name, oklahoma, malum, salutis, webzine, demophon, ulm, moabite,
Nearest to active: facilitated, nami, yugoslavian, dissolves, bcc, ton, saavedra, mandibles,
Nearest to existence: nauvoo, ostracism, pacifists, topkapi, azrael, subordinated, begets, mandible,
Nearest to hold: darin, specify, unspecified, gift, eminem, arbitration, respect, rightarrow,
Nearest to pressure: reactant, temperature, fervently, internal, normally, pools, impartiality, flow,
Nearest to bible: editions, observance, citations, akiva, nevi, tractates, mamre, canonised,
Nearest to joseph: ancient, puente, usk, thyself, fenwick, paraguay, waitakere, harring,
Nearest to gold: loft, oda, silver, refinery, viticulture, calamine, leonis, figs,
Nearest to except: vr, beamed, demobilized, consecrated, match, drops, sandstone, menus,
Epoch 6/10 Iteration: 24100 Avg. Training loss: 3.9266 0.2413 sec/batch
Epoch 6/10 Iteration: 24200 Avg. Training loss: 3.9273 0.2372 sec/batch
Epoch 6/10 Iteration: 24300 Avg. Training loss: 3.8128 0.2355 sec/batch
Epoch 6/10 Iteration: 24400 Avg. Training loss: 3.8917 0.2367 sec/batch
Epoch 6/10 Iteration: 24500 Avg. Training loss: 3.8582 0.2375 sec/batch
Epoch 6/10 Iteration: 24600 Avg. Training loss: 3.8729 0.2372 sec/batch
Epoch 6/10 Iteration: 24700 Avg. Training loss: 3.9094 0.2406 sec/batch
Epoch 6/10 Iteration: 24800 Avg. Training loss: 3.9123 0.2387 sec/batch
Epoch 6/10 Iteration: 24900 Avg. Training loss: 3.8704 0.2415 sec/batch
Epoch 6/10 Iteration: 25000 Avg. Training loss: 3.9261 0.2397 sec/batch
Nearest to time: finish, niqqud, builder, domination, terminates, sanitization, was, the,
Nearest to his: he, prodigy, unsuited, had, chani, homing, governorship, degas,
Nearest to on: the, thirteen, markings, revitalised, quarterly, fragmenting, bitstream, mcneil,
Nearest to many: swooping, candidiasis, known, are, genetics, jit, used, it,
Nearest to was: had, the, ascertaining, time, nls, as, but, spent,
Nearest to that: to, is, committed, the, distant, canonically, chromaticism, it,
Nearest to states: united, faring, america, ocogs, of, univision, gpo, tennessee,
Nearest to known: name, the, oklahoma, malum, salutis, webzine, ulm, moabite,
Nearest to active: facilitated, nami, dissolves, yugoslavian, ton, bcc, saavedra, buzzsaw,
Nearest to existence: nauvoo, ostracism, pacifists, topkapi, azrael, subordinated, begets, intuitionism,
Nearest to hold: darin, eminem, unspecified, specify, casings, arbitration, gift, rightarrow,
Nearest to pressure: reactant, impartiality, pools, fervently, flow, internal, anemometers, temperature,
Nearest to bible: editions, akiva, observance, tractates, canonised, mamre, citations, therapeutae,
Nearest to joseph: ancient, puente, usk, fenwick, paraguay, waitakere, thyself, harring,
Nearest to gold: silver, oda, loft, refinery, viticulture, leonis, figs, calamine,
Nearest to except: vr, beamed, demobilized, match, chooses, hydroxybutyrate, sandstone, drops,
Epoch 6/10 Iteration: 25100 Avg. Training loss: 3.9547 0.2431 sec/batch
Epoch 6/10 Iteration: 25200 Avg. Training loss: 3.8658 0.2589 sec/batch
Epoch 6/10 Iteration: 25300 Avg. Training loss: 3.8818 0.2459 sec/batch
Epoch 6/10 Iteration: 25400 Avg. Training loss: 3.9361 0.2486 sec/batch
Epoch 6/10 Iteration: 25500 Avg. Training loss: 3.9147 0.2467 sec/batch
Epoch 6/10 Iteration: 25600 Avg. Training loss: 3.9057 0.2478 sec/batch
Epoch 6/10 Iteration: 25700 Avg. Training loss: 3.9557 0.2595 sec/batch
Epoch 6/10 Iteration: 25800 Avg. Training loss: 3.8961 0.2507 sec/batch
Epoch 6/10 Iteration: 25900 Avg. Training loss: 3.8899 0.2437 sec/batch
Epoch 6/10 Iteration: 26000 Avg. Training loss: 3.9470 0.2452 sec/batch
Nearest to time: domination, niqqud, finish, builder, terminates, pattie, was, sanitization,
Nearest to his: he, prodigy, governorship, had, unsuited, him, traveled, returned,
Nearest to on: the, thirteen, march, trilateral, poulenc, freedb, quarterly, was,
Nearest to many: candidiasis, swooping, treasure, ensues, are, attacker, used, genetics,
Nearest to was: the, had, ascertaining, of, as, wolfgang, known, time,
Nearest to that: is, to, distant, committed, the, canonically, chromaticism, midland,
Nearest to states: united, america, faring, batoche, ocogs, of, univision, act,
Nearest to known: malum, name, salutis, the, oklahoma, was, ulm, moabite,
Nearest to active: facilitated, nami, saavedra, ton, bcc, yugoslavian, dissolves, comprehensiveness,
Nearest to existence: nauvoo, ostracism, topkapi, azrael, begets, pacifists, fatwas, cyclotron,
Nearest to hold: darin, eminem, specify, unspecified, casings, bullpen, rightarrow, hypothesis,
Nearest to pressure: reactant, fervently, pools, impartiality, internal, temperature, anemometers, newton,
Nearest to bible: editions, observance, akiva, citations, tractates, canonised, mamre, nevi,
Nearest to joseph: ancient, puente, usk, fenwick, waitakere, paraguay, thyself, harring,
Nearest to gold: silver, oda, loft, viticulture, refinery, leonis, figs, calamine,
Nearest to except: vr, beamed, demobilized, match, drops, chooses, consecrated, kara,
Epoch 6/10 Iteration: 26100 Avg. Training loss: 3.9353 0.2518 sec/batch
Epoch 6/10 Iteration: 26200 Avg. Training loss: 3.9339 0.2502 sec/batch
Epoch 6/10 Iteration: 26300 Avg. Training loss: 3.9603 0.2542 sec/batch
Epoch 6/10 Iteration: 26400 Avg. Training loss: 3.8931 0.2516 sec/batch
Epoch 6/10 Iteration: 26500 Avg. Training loss: 3.9010 0.2492 sec/batch
Epoch 6/10 Iteration: 26600 Avg. Training loss: 3.9712 0.2436 sec/batch
Epoch 6/10 Iteration: 26700 Avg. Training loss: 3.8851 0.2370 sec/batch
Epoch 6/10 Iteration: 26800 Avg. Training loss: 3.9700 0.2513 sec/batch
Epoch 6/10 Iteration: 26900 Avg. Training loss: 3.9833 0.2415 sec/batch
Epoch 6/10 Iteration: 27000 Avg. Training loss: 3.9778 0.2410 sec/batch
Nearest to time: domination, terminates, niqqud, sanitization, pattie, finish, builder, in,
Nearest to his: he, prodigy, s, governorship, him, degas, to, unsuited,
Nearest to on: the, thirteen, revitalised, march, trilateral, freedb, fragmenting, various,
Nearest to many: swooping, candidiasis, treasure, ensues, attacker, topping, synagogue, controllers,
Nearest to was: had, the, ascertaining, as, of, known, nls, were,
Nearest to that: the, to, committed, is, distant, chromaticism, hallucinatory, canonically,
Nearest to states: united, america, of, first, tennessee, faring, ocogs, batoche,
Nearest to known: malum, the, oklahoma, moabite, salutis, ulm, demophon, name,
Nearest to active: facilitated, nami, dissolves, yugoslavian, ton, saavedra, comprehensiveness, sippar,
Nearest to existence: nauvoo, ostracism, azrael, topkapi, pacifists, eritis, galbraith, begets,
Nearest to hold: darin, eminem, unspecified, specify, arbitration, dowry, concacaf, bullpen,
Nearest to pressure: fervently, pools, reactant, impartiality, internal, salute, anemometers, air,
Nearest to bible: editions, tractates, sevenfold, observance, citations, mamre, marzipan, nevi,
Nearest to joseph: ancient, puente, usk, fenwick, waitakere, harring, paraguay, successions,
Nearest to gold: silver, oda, loft, viticulture, refinery, figs, leonis, calamine,
Nearest to except: demobilized, vr, beamed, match, drops, chooses, kara, consecrated,
Epoch 6/10 Iteration: 27100 Avg. Training loss: 3.8927 0.2482 sec/batch
Epoch 6/10 Iteration: 27200 Avg. Training loss: 3.8408 0.2448 sec/batch
Epoch 6/10 Iteration: 27300 Avg. Training loss: 3.9184 0.2380 sec/batch
Epoch 6/10 Iteration: 27400 Avg. Training loss: 3.7978 0.2365 sec/batch
Epoch 6/10 Iteration: 27500 Avg. Training loss: 3.9277 0.2358 sec/batch
Epoch 6/10 Iteration: 27600 Avg. Training loss: 3.8906 0.2416 sec/batch
Epoch 6/10 Iteration: 27700 Avg. Training loss: 3.8824 0.2455 sec/batch
Epoch 7/10 Iteration: 27800 Avg. Training loss: 3.9385 0.1037 sec/batch
Epoch 7/10 Iteration: 27900 Avg. Training loss: 3.8681 0.2372 sec/batch
Epoch 7/10 Iteration: 28000 Avg. Training loss: 3.8574 0.2352 sec/batch
Nearest to time: domination, niqqud, pattie, finish, terminates, in, sanitization, builder,
Nearest to his: he, prodigy, unsuited, degas, to, s, him, chani,
Nearest to on: the, thirteen, march, poulenc, epimetheus, fragmenting, trilateral, freedb,
Nearest to many: swooping, candidiasis, treasure, dss, used, are, particularly, ferrous,
Nearest to was: had, the, ascertaining, as, reconstructed, spent, but, of,
Nearest to that: committed, to, the, chromaticism, hallucinatory, is, millenarianism, musicological,
Nearest to states: united, america, faring, ocogs, tennessee, of, batoche, liechtenstein,
Nearest to known: oklahoma, malum, the, demophon, moabite, their, salutis, webzine,
Nearest to active: facilitated, nami, dissolves, yugoslavian, saavedra, mandibles, teotihuacan, ton,
Nearest to existence: nauvoo, ostracism, pacifists, azrael, topkapi, intuitionism, primeval, begets,
Nearest to hold: darin, specify, eminem, unspecified, hypothesis, arian, respect, gift,
Nearest to pressure: reactant, fervently, impartiality, temperature, flow, anemometers, pools, internal,
Nearest to bible: editions, observance, sevenfold, tractates, marzipan, citations, canonised, therapeutae,
Nearest to joseph: ancient, puente, usk, fenwick, waitakere, mendelssohn, harring, successions,
Nearest to gold: silver, loft, oda, figs, viticulture, refinery, calamine, leonis,
Nearest to except: vr, beamed, demobilized, match, drops, chooses, kara, devolution,
Epoch 7/10 Iteration: 28100 Avg. Training loss: 3.8479 0.2390 sec/batch
Epoch 7/10 Iteration: 28200 Avg. Training loss: 3.8528 0.2348 sec/batch
Epoch 7/10 Iteration: 28300 Avg. Training loss: 3.8525 0.2350 sec/batch
Epoch 7/10 Iteration: 28400 Avg. Training loss: 3.8352 0.2334 sec/batch
Epoch 7/10 Iteration: 28500 Avg. Training loss: 3.8005 0.2330 sec/batch
Epoch 7/10 Iteration: 28600 Avg. Training loss: 3.8723 0.2341 sec/batch
Epoch 7/10 Iteration: 28700 Avg. Training loss: 3.9146 0.2318 sec/batch
Epoch 7/10 Iteration: 28800 Avg. Training loss: 3.9034 0.2337 sec/batch
Epoch 7/10 Iteration: 28900 Avg. Training loss: 3.7721 0.2353 sec/batch
Epoch 7/10 Iteration: 29000 Avg. Training loss: 3.8541 0.2332 sec/batch
Nearest to time: domination, was, the, finish, niqqud, mariano, terminates, in,
Nearest to his: he, prodigy, unsuited, homing, degas, returned, him, mccartney,
Nearest to on: the, various, march, thirteen, main, quarterly, of, forecasts,
Nearest to many: known, used, are, dss, particularly, candidiasis, swooping, attacker,
Nearest to was: had, the, but, by, of, time, reconstructed, were,
Nearest to that: to, is, the, committed, it, distant, arnage, and,
Nearest to states: united, of, america, faring, is, liechtenstein, act, ocogs,
Nearest to known: malum, oklahoma, the, many, name, their, both, demophon,
Nearest to active: facilitated, nami, dissolves, yugoslavian, sippar, teotihuacan, mandibles, jure,
Nearest to existence: nauvoo, ostracism, pacifists, azrael, topkapi, knapsack, cyclotron, nuanced,
Nearest to hold: darin, specify, eminem, unspecified, gift, casings, rightarrow, logbook,
Nearest to pressure: pools, impartiality, neuroanatomy, reactant, fervently, volleyball, anemometers, flow,
Nearest to bible: editions, observance, tractates, therapeutae, nevi, mamre, transcribed, tocantins,
Nearest to joseph: ancient, puente, usk, waitakere, fenwick, successions, nadab, paraguay,
Nearest to gold: silver, oda, loft, calamine, viticulture, refinery, figs, leonis,
Nearest to except: vr, beamed, demobilized, kara, match, arboreal, consecrated, chooses,
Epoch 7/10 Iteration: 29100 Avg. Training loss: 3.8793 0.2365 sec/batch
Epoch 7/10 Iteration: 29200 Avg. Training loss: 3.8099 0.2347 sec/batch
Epoch 7/10 Iteration: 29300 Avg. Training loss: 3.8493 0.2329 sec/batch
Epoch 7/10 Iteration: 29400 Avg. Training loss: 3.8781 0.2343 sec/batch
Epoch 7/10 Iteration: 29500 Avg. Training loss: 3.8436 0.2356 sec/batch
Epoch 7/10 Iteration: 29600 Avg. Training loss: 3.8890 0.2380 sec/batch
Epoch 7/10 Iteration: 29700 Avg. Training loss: 3.9012 0.2344 sec/batch
Epoch 7/10 Iteration: 29800 Avg. Training loss: 3.8862 0.2367 sec/batch
Epoch 7/10 Iteration: 29900 Avg. Training loss: 3.8595 0.2356 sec/batch
Epoch 7/10 Iteration: 30000 Avg. Training loss: 3.8662 0.2370 sec/batch
Nearest to time: the, terminates, in, was, domination, finish, niqqud, beyond,
Nearest to his: he, s, prodigy, him, returned, had, unsuited, degas,
Nearest to on: the, thirteen, march, trilateral, freedb, various, poulenc, quarterly,
Nearest to many: known, used, are, candidiasis, attacker, dss, particularly, nnn,
Nearest to was: the, had, by, of, but, were, as, reconstructed,
Nearest to that: to, the, is, it, committed, and, distant, for,
Nearest to states: united, america, faring, liechtenstein, of, batoche, first, act,
Nearest to known: malum, both, many, oklahoma, the, name, their, was,
Nearest to active: facilitated, nami, dissolves, saavedra, monasticism, sippar, oxidants, comprehensiveness,
Nearest to existence: nauvoo, ostracism, azrael, cyclotron, creator, topkapi, pacifists, knapsack,
Nearest to hold: darin, eminem, specify, hypothesis, unspecified, logbook, casings, arian,
Nearest to pressure: pools, impartiality, fervently, internal, reactant, berbers, volleyball, neuroanatomy,
Nearest to bible: editions, observance, mullis, reaches, tractates, tocantins, therapeutae, citations,
Nearest to joseph: ancient, puente, usk, fenwick, waitakere, breadth, paraguay, harring,
Nearest to gold: silver, oda, loft, calamine, viticulture, refinery, courbet, leonis,
Nearest to except: vr, demobilized, arboreal, beamed, match, consecrated, kara, chooses,
Epoch 7/10 Iteration: 30100 Avg. Training loss: 3.8768 0.2386 sec/batch
Epoch 7/10 Iteration: 30200 Avg. Training loss: 3.9168 0.2345 sec/batch
Epoch 7/10 Iteration: 30300 Avg. Training loss: 3.8755 0.2340 sec/batch
Epoch 7/10 Iteration: 30400 Avg. Training loss: 3.8257 0.2362 sec/batch
Epoch 7/10 Iteration: 30500 Avg. Training loss: 3.8927 0.2398 sec/batch
Epoch 7/10 Iteration: 30600 Avg. Training loss: 3.8856 0.2399 sec/batch
Epoch 7/10 Iteration: 30700 Avg. Training loss: 3.8692 0.2479 sec/batch
Epoch 7/10 Iteration: 30800 Avg. Training loss: 3.8718 0.2390 sec/batch
Epoch 7/10 Iteration: 30900 Avg. Training loss: 3.8862 0.2394 sec/batch
Epoch 7/10 Iteration: 31000 Avg. Training loss: 3.8617 0.2400 sec/batch
Nearest to time: domination, terminates, niqqud, was, beyond, in, the, eight,
Nearest to his: he, him, s, prodigy, returned, governorship, unsuited, degas,
Nearest to on: the, evapotranspiration, various, trilateral, march, an, forecasts, was,
Nearest to many: used, candidiasis, known, ferrous, particularly, dss, are, untouchability,
Nearest to was: had, the, as, by, of, were, but, and,
Nearest to that: the, to, is, and, distant, committed, in, it,
Nearest to states: united, america, first, faring, batoche, of, ocogs, reinforces,
Nearest to known: malum, both, the, their, oklahoma, was, by, salutis,
Nearest to active: facilitated, oxidants, nami, dissolves, monasticism, saavedra, volcanism, ingestion,
Nearest to existence: nauvoo, ostracism, azrael, mandible, heist, topkapi, knapsack, cyclotron,
Nearest to hold: specify, darin, hypothesis, eminem, arian, unspecified, leader, casings,
Nearest to pressure: fervently, internal, impartiality, pools, berbers, volleyball, reactant, romanorum,
Nearest to bible: editions, transcribed, tractates, observance, reaches, mullis, exhorted, favoritism,
Nearest to joseph: ancient, puente, usk, waitakere, fenwick, breadth, paraguay, successions,
Nearest to gold: silver, oda, loft, viticulture, calamine, refinery, figs, leonis,
Nearest to except: vr, demobilized, beamed, match, arboreal, kara, chooses, consecrated,
Epoch 7/10 Iteration: 31100 Avg. Training loss: 3.8397 0.2413 sec/batch
Epoch 7/10 Iteration: 31200 Avg. Training loss: 3.8904 0.2371 sec/batch
Epoch 7/10 Iteration: 31300 Avg. Training loss: 3.8855 0.2396 sec/batch
Epoch 7/10 Iteration: 31400 Avg. Training loss: 3.9590 0.2397 sec/batch
Epoch 7/10 Iteration: 31500 Avg. Training loss: 3.9379 0.2399 sec/batch
Epoch 7/10 Iteration: 31600 Avg. Training loss: 3.9327 0.2396 sec/batch
Epoch 7/10 Iteration: 31700 Avg. Training loss: 3.8710 0.2394 sec/batch
Epoch 7/10 Iteration: 31800 Avg. Training loss: 3.8294 0.2377 sec/batch
Epoch 7/10 Iteration: 31900 Avg. Training loss: 3.8633 0.2380 sec/batch
Epoch 7/10 Iteration: 32000 Avg. Training loss: 3.7765 0.2407 sec/batch
Nearest to time: domination, the, in, finish, terminates, was, niqqud, caliphates,
Nearest to his: he, him, prodigy, s, returned, degas, to, unsuited,
Nearest to on: the, trilateral, march, an, evapotranspiration, same, thirteen, poulenc,
Nearest to many: used, known, swooping, nnn, candidiasis, ensues, particularly, ferrous,
Nearest to was: the, had, as, were, and, of, by, but,
Nearest to that: the, to, is, committed, and, in, it, distant,
Nearest to states: united, america, first, of, faring, liechtenstein, batoche, tennessee,
Nearest to known: malum, oklahoma, both, the, their, tonga, many, demophon,
Nearest to active: facilitated, dissolves, nami, yugoslavian, organization, sippar, jure, volcanism,
Nearest to existence: nauvoo, azrael, knapsack, ostracism, mandible, topkapi, pacifists, decompose,
Nearest to hold: darin, leader, eminem, hypothesis, unspecified, specify, respect, casings,
Nearest to pressure: fervently, volleyball, impartiality, pools, internal, berbers, salute, reactant,
Nearest to bible: editions, sevenfold, observance, mullis, tractates, transcribed, reaches, exhorted,
Nearest to joseph: ancient, puente, usk, fenwick, waitakere, successions, breadth, paraguay,
Nearest to gold: silver, oda, loft, calamine, viticulture, refinery, leonis, figs,
Nearest to except: vr, demobilized, beamed, match, kara, pembroke, arboreal, chooses,
Epoch 7/10 Iteration: 32100 Avg. Training loss: 3.8810 0.2469 sec/batch
Epoch 7/10 Iteration: 32200 Avg. Training loss: 3.8600 0.2358 sec/batch
Epoch 7/10 Iteration: 32300 Avg. Training loss: 3.8462 0.2355 sec/batch
Epoch 8/10 Iteration: 32400 Avg. Training loss: 3.8949 0.0416 sec/batch
Epoch 8/10 Iteration: 32500 Avg. Training loss: 3.8725 0.2339 sec/batch
Epoch 8/10 Iteration: 32600 Avg. Training loss: 3.8260 0.2336 sec/batch
Epoch 8/10 Iteration: 32700 Avg. Training loss: 3.8425 0.2329 sec/batch
Epoch 8/10 Iteration: 32800 Avg. Training loss: 3.8387 0.2333 sec/batch
Epoch 8/10 Iteration: 32900 Avg. Training loss: 3.8004 0.2333 sec/batch
Epoch 8/10 Iteration: 33000 Avg. Training loss: 3.8373 0.2368 sec/batch
Nearest to time: domination, was, in, the, beyond, paleontological, terminates, after,
Nearest to his: he, him, returned, s, son, to, had, homing,
Nearest to on: the, was, an, apollo, thirteen, march, same, august,
Nearest to many: used, known, particularly, ensues, candidiasis, swooping, it, are,
Nearest to was: the, had, as, but, by, of, and, on,
Nearest to that: to, the, in, and, committed, is, it, which,
Nearest to states: united, first, america, faring, batoche, of, liechtenstein, tennessee,
Nearest to known: the, malum, both, name, oklahoma, their, was, many,
Nearest to active: facilitated, volcanism, nami, dissolves, oxidants, promotes, mandibles, sippar,
Nearest to existence: nauvoo, azrael, heist, pacifists, topkapi, knapsack, eritis, nuanced,
Nearest to hold: darin, hypothesis, unspecified, leader, specify, casings, eminem, arian,
Nearest to pressure: pools, volleyball, flow, berbers, impartiality, neuroanatomy, internal, fervently,
Nearest to bible: editions, sevenfold, transcribed, observance, masoretes, reaches, mullis, therapeutae,
Nearest to joseph: ancient, puente, usk, waitakere, fenwick, successions, breadth, peart,
Nearest to gold: silver, oda, loft, calamine, refinery, leonis, viticulture, sacred,
Nearest to except: vr, beamed, demobilized, arboreal, kara, match, chooses, shipped,
Epoch 8/10 Iteration: 33100 Avg. Training loss: 3.8014 0.2359 sec/batch
Epoch 8/10 Iteration: 33200 Avg. Training loss: 3.8114 0.2317 sec/batch
Epoch 8/10 Iteration: 33300 Avg. Training loss: 3.8347 0.2351 sec/batch
Epoch 8/10 Iteration: 33400 Avg. Training loss: 3.8919 0.2380 sec/batch
Epoch 8/10 Iteration: 33500 Avg. Training loss: 3.7934 0.2350 sec/batch
Epoch 8/10 Iteration: 33600 Avg. Training loss: 3.8226 0.2373 sec/batch
Epoch 8/10 Iteration: 33700 Avg. Training loss: 3.8052 0.2347 sec/batch
Epoch 8/10 Iteration: 33800 Avg. Training loss: 3.7859 0.2334 sec/batch
Epoch 8/10 Iteration: 33900 Avg. Training loss: 3.8300 0.2338 sec/batch
Epoch 8/10 Iteration: 34000 Avg. Training loss: 3.8415 0.2376 sec/batch
Nearest to time: domination, the, in, was, after, niqqud, paleontological, finish,
Nearest to his: he, him, to, returned, had, son, s, prodigy,
Nearest to on: the, was, same, various, thirteen, revitalised, an, trilateral,
Nearest to many: used, particularly, candidiasis, known, are, as, it, swooping,
Nearest to was: had, the, by, as, but, though, were, on,
Nearest to that: to, the, is, and, it, which, in, committed,
Nearest to states: united, of, first, america, faring, act, ocogs, usa,
Nearest to known: both, the, name, oklahoma, malum, their, was, many,
Nearest to active: facilitated, volcanism, dissolves, oxidants, nami, facilitate, mandibles, sippar,
Nearest to existence: nauvoo, pacifists, heist, azrael, creator, bitterness, nuanced, knapsack,
Nearest to hold: specify, unspecified, hypothesis, eminem, darin, gift, casings, leader,
Nearest to pressure: pools, impartiality, reactant, fervently, sublimation, berbers, neuroanatomy, volleyball,
Nearest to bible: editions, sevenfold, transcribed, reaches, masoretes, mullis, therapeutae, mamre,
Nearest to joseph: ancient, puente, usk, waitakere, fenwick, successions, breadth, harring,
Nearest to gold: silver, oda, loft, calamine, viticulture, refinery, leonis, patrimony,
Nearest to except: vr, demobilized, chooses, beamed, match, arboreal, kara, they,
Epoch 8/10 Iteration: 34100 Avg. Training loss: 3.8063 0.2394 sec/batch
Epoch 8/10 Iteration: 34200 Avg. Training loss: 3.8290 0.2359 sec/batch
Epoch 8/10 Iteration: 34300 Avg. Training loss: 3.8962 0.2356 sec/batch
Epoch 8/10 Iteration: 34400 Avg. Training loss: 3.8690 0.2333 sec/batch
Epoch 8/10 Iteration: 34500 Avg. Training loss: 3.8142 0.2358 sec/batch
Epoch 8/10 Iteration: 34600 Avg. Training loss: 3.8194 0.2360 sec/batch
Epoch 8/10 Iteration: 34700 Avg. Training loss: 3.8396 0.2355 sec/batch
Epoch 8/10 Iteration: 34800 Avg. Training loss: 3.8562 0.2388 sec/batch
Epoch 8/10 Iteration: 34900 Avg. Training loss: 3.8366 0.2331 sec/batch
Epoch 8/10 Iteration: 35000 Avg. Training loss: 3.8647 0.2340 sec/batch
Nearest to time: domination, the, in, terminates, paleontological, after, beyond, was,
Nearest to his: he, s, him, prodigy, degas, returned, career, son,
Nearest to on: the, was, is, same, which, trilateral, by, an,
Nearest to many: used, candidiasis, particularly, swooping, known, ensues, are, untouchability,
Nearest to was: the, had, which, as, of, by, though, on,
Nearest to that: to, is, the, for, it, and, in, which,
Nearest to states: united, first, of, america, faring, act, usa, batoche,
Nearest to known: both, name, the, oklahoma, malum, was, by, their,
Nearest to active: volcanism, facilitated, dissolves, nami, roadmap, oxidants, facilitate, sippar,
Nearest to existence: nauvoo, pacifists, bitterness, heist, subordinated, azrael, cyclotron, mandible,
Nearest to hold: hypothesis, casings, specify, darin, eminem, unspecified, respect, leader,
Nearest to pressure: pools, impartiality, fervently, volleyball, berbers, internal, sublimation, reactant,
Nearest to bible: editions, sevenfold, reaches, transcribed, mullis, favoritism, masoretes, ipf,
Nearest to joseph: ancient, puente, usk, fenwick, waitakere, breadth, paraguay, osprey,
Nearest to gold: silver, oda, loft, viticulture, leonis, calamine, refinery, courbet,
Nearest to except: vr, demobilized, match, chooses, beamed, kara, arboreal, exited,
Epoch 8/10 Iteration: 35100 Avg. Training loss: 3.8440 0.2368 sec/batch
Epoch 8/10 Iteration: 35200 Avg. Training loss: 3.8630 0.2347 sec/batch
Epoch 8/10 Iteration: 35300 Avg. Training loss: 3.8428 0.2342 sec/batch
Epoch 8/10 Iteration: 35400 Avg. Training loss: 3.8773 0.2341 sec/batch
Epoch 8/10 Iteration: 35500 Avg. Training loss: 3.8522 0.2318 sec/batch
Epoch 8/10 Iteration: 35600 Avg. Training loss: 3.8374 0.2355 sec/batch
Epoch 8/10 Iteration: 35700 Avg. Training loss: 3.8089 0.2332 sec/batch
Epoch 8/10 Iteration: 35800 Avg. Training loss: 3.7903 0.2308 sec/batch
Epoch 8/10 Iteration: 35900 Avg. Training loss: 3.8653 0.2352 sec/batch
Epoch 8/10 Iteration: 36000 Avg. Training loss: 3.8948 0.2342 sec/batch
Nearest to time: in, the, domination, eight, was, terminates, paleontological, pattie,
Nearest to his: he, him, to, s, returned, career, prodigy, had,
Nearest to on: the, was, of, in, by, a, an, is,
Nearest to many: candidiasis, used, particularly, swooping, most, quisqueya, ensues, untouchability,
Nearest to was: had, the, of, as, and, by, which, in,
Nearest to that: the, to, and, for, in, it, distant, is,
Nearest to states: united, first, of, america, faring, act, batoche, madison,
Nearest to known: both, malum, the, oklahoma, name, salutis, by, was,
Nearest to active: volcanism, facilitated, dissolves, nami, facilitate, oxidants, roadmap, sippar,
Nearest to existence: nauvoo, azrael, heist, topkapi, outcroppings, pacifists, mandible, bitterness,
Nearest to hold: darin, hypothesis, dowry, eminem, casings, specify, leader, respect,
Nearest to pressure: fervently, impartiality, pools, internal, volleyball, berbers, air, sublimation,
Nearest to bible: sevenfold, reaches, transcribed, editions, mullis, ipf, masoretes, exhorted,
Nearest to joseph: puente, ancient, fenwick, usk, physician, waitakere, colonies, breadth,
Nearest to gold: silver, oda, loft, viticulture, calamine, refinery, leonis, courbet,
Nearest to except: demobilized, match, vr, kara, beamed, word, they, chooses,
Epoch 8/10 Iteration: 36100 Avg. Training loss: 3.9333 0.2354 sec/batch
Epoch 8/10 Iteration: 36200 Avg. Training loss: 3.9037 0.2329 sec/batch
Epoch 8/10 Iteration: 36300 Avg. Training loss: 3.8498 0.2314 sec/batch
Epoch 8/10 Iteration: 36400 Avg. Training loss: 3.8151 0.2336 sec/batch
Epoch 8/10 Iteration: 36500 Avg. Training loss: 3.8266 0.2303 sec/batch
Epoch 8/10 Iteration: 36600 Avg. Training loss: 3.7744 0.2337 sec/batch
Epoch 8/10 Iteration: 36700 Avg. Training loss: 3.8451 0.2330 sec/batch
Epoch 8/10 Iteration: 36800 Avg. Training loss: 3.8576 0.2345 sec/batch
Epoch 8/10 Iteration: 36900 Avg. Training loss: 3.8250 0.2309 sec/batch
Epoch 8/10 Iteration: 37000 Avg. Training loss: 3.8612 0.2310 sec/batch
Nearest to time: in, domination, the, paleontological, niqqud, sync, pattie, terminates,
Nearest to his: he, him, son, degas, returned, father, career, s,
Nearest to on: the, was, is, same, various, march, an, of,
Nearest to many: most, used, are, particularly, swooping, candidiasis, known, quisqueya,
Nearest to was: had, the, as, and, by, of, on, but,
Nearest to that: the, to, for, is, it, and, in, committed,
Nearest to states: united, first, of, america, faring, kacl, ocogs, unrealized,
Nearest to known: both, the, malum, oklahoma, their, by, salutis, name,
Nearest to active: volcanism, facilitated, dissolves, oxidants, nami, saavedra, facilitate, promotes,
Nearest to existence: nauvoo, eac, outcroppings, heist, subordinated, pacifists, azrael, mandible,
Nearest to hold: hypothesis, darin, casings, dowry, specify, respect, eminem, unspecified,
Nearest to pressure: volleyball, berbers, fervently, pools, internal, impartiality, sublimation, air,
Nearest to bible: sevenfold, reaches, editions, transcribed, mullis, ipf, insights, masoretes,
Nearest to joseph: puente, ancient, fenwick, usk, waitakere, physician, colonies, harring,
Nearest to gold: silver, oda, loft, calamine, viticulture, waiver, leonis, refinery,
Nearest to except: demobilized, vr, they, match, word, kara, beamed, shipped,
Epoch 9/10 Iteration: 37100 Avg. Training loss: 3.8369 0.2151 sec/batch
Epoch 9/10 Iteration: 37200 Avg. Training loss: 3.8253 0.2322 sec/batch
Epoch 9/10 Iteration: 37300 Avg. Training loss: 3.8036 0.2365 sec/batch
Epoch 9/10 Iteration: 37400 Avg. Training loss: 3.8371 0.2334 sec/batch
Epoch 9/10 Iteration: 37500 Avg. Training loss: 3.7779 0.2323 sec/batch
Epoch 9/10 Iteration: 37600 Avg. Training loss: 3.8089 0.2356 sec/batch
Epoch 9/10 Iteration: 37700 Avg. Training loss: 3.7695 0.2351 sec/batch
Epoch 9/10 Iteration: 37800 Avg. Training loss: 3.8368 0.2326 sec/batch
Epoch 9/10 Iteration: 37900 Avg. Training loss: 3.8573 0.2332 sec/batch
Epoch 9/10 Iteration: 38000 Avg. Training loss: 3.8372 0.2358 sec/batch
Nearest to time: in, the, was, domination, sync, niqqud, paleontological, scalemajor,
Nearest to his: he, him, returned, career, s, to, had, son,
Nearest to on: the, was, by, of, which, an, is, various,
Nearest to many: used, most, particularly, are, candidiasis, known, some, swooping,
Nearest to was: the, had, by, as, and, but, on, were,
Nearest to that: the, to, it, is, and, in, a, committed,
Nearest to states: united, first, of, faring, america, kacl, ocogs, cw,
Nearest to known: both, the, by, name, malum, oklahoma, in, their,
Nearest to active: volcanism, facilitated, dissolves, saavedra, nami, facilitate, roadmap, oxidants,
Nearest to existence: nauvoo, heist, azrael, eac, outcroppings, bitterness, bolland, mandible,
Nearest to hold: dowry, specify, darin, casings, hypothesis, unspecified, celebrate, favor,
Nearest to pressure: volleyball, berbers, pools, fervently, sublimation, impartiality, internal, salute,
Nearest to bible: sevenfold, editions, transcribed, reaches, masoretes, ipf, mamre, redactor,
Nearest to joseph: puente, ancient, usk, fenwick, colloquy, thyself, meir, waitakere,
Nearest to gold: silver, oda, calamine, loft, viticulture, leonis, figs, sacred,
Nearest to except: they, vr, demobilized, beamed, match, shipped, arboreal, in,
Epoch 9/10 Iteration: 38100 Avg. Training loss: 3.8232 0.2357 sec/batch
Epoch 9/10 Iteration: 38200 Avg. Training loss: 3.7380 0.2361 sec/batch
Epoch 9/10 Iteration: 38300 Avg. Training loss: 3.8153 0.2340 sec/batch
Epoch 9/10 Iteration: 38400 Avg. Training loss: 3.7407 0.2335 sec/batch
Epoch 9/10 Iteration: 38500 Avg. Training loss: 3.8186 0.2328 sec/batch
Epoch 9/10 Iteration: 38600 Avg. Training loss: 3.8392 0.2333 sec/batch
Epoch 9/10 Iteration: 38700 Avg. Training loss: 3.8232 0.2334 sec/batch
Epoch 9/10 Iteration: 38800 Avg. Training loss: 3.8006 0.2339 sec/batch
Epoch 9/10 Iteration: 38900 Avg. Training loss: 3.8466 0.2317 sec/batch
Epoch 9/10 Iteration: 39000 Avg. Training loss: 3.8779 0.2343 sec/batch
Nearest to time: in, the, domination, sync, paleontological, was, terminates, scalemajor,
Nearest to his: he, him, s, returned, career, degas, had, prodigy,
Nearest to on: the, was, is, by, a, which, and, various,
Nearest to many: most, used, particularly, known, are, as, candidiasis, some,
Nearest to was: the, had, by, as, and, of, which, on,
Nearest to that: to, the, is, it, for, and, in, which,
Nearest to states: united, first, of, america, faring, unrealized, kacl, is,
Nearest to known: both, the, by, was, many, name, malum, their,
Nearest to active: volcanism, dissolves, facilitated, saavedra, oxidants, roadmap, nami, facilitate,
Nearest to existence: nauvoo, heist, azrael, eac, creator, takings, bitterness, bolland,
Nearest to hold: casings, hypothesis, dowry, eminem, favor, darin, blackberry, specify,
Nearest to pressure: volleyball, pools, berbers, sublimation, internal, impartiality, fervently, shatter,
Nearest to bible: sevenfold, editions, reaches, transcribed, masoretes, ipf, redactor, mullis,
Nearest to joseph: puente, ancient, fenwick, usk, peart, colloquy, thyself, bundling,
Nearest to gold: silver, oda, loft, viticulture, calamine, leonis, courbet, refinery,
Nearest to except: match, vr, arboreal, demobilized, they, apportioned, humphries, shipped,
Epoch 9/10 Iteration: 39100 Avg. Training loss: 3.7700 0.2381 sec/batch
Epoch 9/10 Iteration: 39200 Avg. Training loss: 3.8347 0.2330 sec/batch
Epoch 9/10 Iteration: 39300 Avg. Training loss: 3.8003 0.2328 sec/batch
Epoch 9/10 Iteration: 39400 Avg. Training loss: 3.8206 0.2331 sec/batch
Epoch 9/10 Iteration: 39500 Avg. Training loss: 3.8188 0.2333 sec/batch
Epoch 9/10 Iteration: 39600 Avg. Training loss: 3.8364 0.2331 sec/batch
Epoch 9/10 Iteration: 39700 Avg. Training loss: 3.8213 0.2331 sec/batch
Epoch 9/10 Iteration: 39800 Avg. Training loss: 3.8236 0.2323 sec/batch
Epoch 9/10 Iteration: 39900 Avg. Training loss: 3.8277 0.2322 sec/batch
Epoch 9/10 Iteration: 40000 Avg. Training loss: 3.8296 0.2331 sec/batch
Nearest to time: in, the, domination, was, sync, beyond, terminates, paleontological,
Nearest to his: he, him, returned, degas, son, career, prodigy, father,
Nearest to on: the, is, was, which, of, by, an, and,
Nearest to many: used, most, candidiasis, as, particularly, quisqueya, untouchability, ferrous,
Nearest to was: the, had, as, by, of, and, were, became,
Nearest to that: the, to, it, is, in, for, and, which,
Nearest to states: united, first, america, of, faring, act, liechtenstein, batoche,
Nearest to known: both, the, was, their, by, malum, name, many,
Nearest to active: volcanism, dissolves, facilitated, saavedra, oxidants, roadmap, facilitate, jure,
Nearest to existence: nauvoo, heist, azrael, eac, bitterness, outcroppings, eritis, topkapi,
Nearest to hold: casings, eminem, hypothesis, dowry, blackberry, specify, unspecified, darin,
Nearest to pressure: volleyball, pools, internal, sublimation, berbers, fervently, impartiality, schismatic,
Nearest to bible: sevenfold, reaches, transcribed, editions, masoretes, exhorted, ipf, redactor,
Nearest to joseph: puente, ancient, fenwick, colloquy, waitakere, usk, peart, thyself,
Nearest to gold: silver, oda, loft, calamine, viticulture, risings, leonis, waiver,
Nearest to except: they, demobilized, match, arboreal, shipped, apportioned, humphries, vr,
Epoch 9/10 Iteration: 40100 Avg. Training loss: 3.8472 0.2350 sec/batch
Epoch 9/10 Iteration: 40200 Avg. Training loss: 3.8225 0.2331 sec/batch
Epoch 9/10 Iteration: 40300 Avg. Training loss: 3.8110 0.2339 sec/batch
Epoch 9/10 Iteration: 40400 Avg. Training loss: 3.7972 0.2334 sec/batch
Epoch 9/10 Iteration: 40500 Avg. Training loss: 3.8632 0.2356 sec/batch
Epoch 9/10 Iteration: 40600 Avg. Training loss: 3.8115 0.2350 sec/batch
Epoch 9/10 Iteration: 40700 Avg. Training loss: 3.9324 0.2365 sec/batch
Epoch 9/10 Iteration: 40800 Avg. Training loss: 3.8708 0.2339 sec/batch
Epoch 9/10 Iteration: 40900 Avg. Training loss: 3.8950 0.2346 sec/batch
Epoch 9/10 Iteration: 41000 Avg. Training loss: 3.8025 0.2346 sec/batch
Nearest to time: in, domination, the, sync, beyond, paleontological, pattie, after,
Nearest to his: he, him, returned, career, had, to, degas, s,
Nearest to on: the, was, of, an, and, which, march, by,
Nearest to many: most, used, particularly, popular, ferrous, quisqueya, some, candidiasis,
Nearest to was: had, the, as, by, on, and, of, which,
Nearest to that: the, to, is, and, it, for, which, by,
Nearest to states: united, first, america, of, faring, kacl, tennessee, jannaeus,
Nearest to known: both, malum, oklahoma, was, their, name, the, most,
Nearest to active: volcanism, dissolves, facilitated, oxidants, saavedra, sippar, jure, nami,
Nearest to existence: nauvoo, outcroppings, heist, eac, azrael, bitterness, strontium, topkapi,
Nearest to hold: casings, hypothesis, dowry, eminem, darin, favor, unspecified, splintering,
Nearest to pressure: pools, internal, berbers, volleyball, sublimation, fervently, impartiality, air,
Nearest to bible: sevenfold, editions, reaches, masoretes, transcribed, exhorted, ipf, brooklyn,
Nearest to joseph: puente, ancient, fenwick, colloquy, mendelssohn, colonies, usk, waitakere,
Nearest to gold: silver, oda, loft, calamine, viticulture, waiver, refinery, leonis,
Nearest to except: they, demobilized, apportioned, arboreal, drops, humphries, shipped, match,
Epoch 9/10 Iteration: 41100 Avg. Training loss: 3.7702 0.2390 sec/batch
Epoch 9/10 Iteration: 41200 Avg. Training loss: 3.8182 0.2329 sec/batch
Epoch 9/10 Iteration: 41300 Avg. Training loss: 3.7339 0.2345 sec/batch
Epoch 9/10 Iteration: 41400 Avg. Training loss: 3.8326 0.2342 sec/batch
Epoch 9/10 Iteration: 41500 Avg. Training loss: 3.8115 0.2347 sec/batch
Epoch 9/10 Iteration: 41600 Avg. Training loss: 3.8464 0.2328 sec/batch
Epoch 10/10 Iteration: 41700 Avg. Training loss: 3.8199 0.1539 sec/batch
Epoch 10/10 Iteration: 41800 Avg. Training loss: 3.8043 0.2336 sec/batch
Epoch 10/10 Iteration: 41900 Avg. Training loss: 3.7495 0.2342 sec/batch
Epoch 10/10 Iteration: 42000 Avg. Training loss: 3.8447 0.2360 sec/batch
Nearest to time: paleontological, the, in, domination, was, sync, pattie, beyond,
Nearest to his: he, him, to, father, son, returned, had, s,
Nearest to on: the, was, of, which, an, and, march, by,
Nearest to many: used, as, most, particularly, some, candidiasis, known, are,
Nearest to was: had, the, as, of, and, by, on, but,
Nearest to that: to, the, is, and, it, for, in, which,
Nearest to states: united, first, of, america, tennessee, faring, mee, jannaeus,
Nearest to known: both, the, by, in, was, malum, oklahoma, as,
Nearest to active: volcanism, dissolves, oxidants, facilitated, nami, saavedra, suffocating, sippar,
Nearest to existence: heist, nauvoo, azrael, eac, outcroppings, creator, intuitionism, strontium,
Nearest to hold: hypothesis, dowry, casings, darin, favor, eminem, blackberry, gift,
Nearest to pressure: pools, volleyball, berbers, internal, impartiality, sublimation, fervently, neuroanatomy,
Nearest to bible: sevenfold, editions, reaches, masoretes, transcribed, redactor, therapeutae, exhorted,
Nearest to joseph: puente, ancient, fenwick, mendelssohn, waitakere, colloquy, colonies, physician,
Nearest to gold: silver, oda, loft, calamine, leonis, risings, viticulture, waiver,
Nearest to except: they, apportioned, humphries, shipped, demobilized, vr, sundry, apart,
Epoch 10/10 Iteration: 42100 Avg. Training loss: 3.8001 0.2386 sec/batch
Epoch 10/10 Iteration: 42200 Avg. Training loss: 3.7972 0.2336 sec/batch
Epoch 10/10 Iteration: 42300 Avg. Training loss: 3.7627 0.2353 sec/batch
Epoch 10/10 Iteration: 42400 Avg. Training loss: 3.8032 0.2355 sec/batch
Epoch 10/10 Iteration: 42500 Avg. Training loss: 3.7888 0.2337 sec/batch
Epoch 10/10 Iteration: 42600 Avg. Training loss: 3.8010 0.2343 sec/batch
Epoch 10/10 Iteration: 42700 Avg. Training loss: 3.8619 0.2332 sec/batch
Epoch 10/10 Iteration: 42800 Avg. Training loss: 3.7077 0.2330 sec/batch
Epoch 10/10 Iteration: 42900 Avg. Training loss: 3.7880 0.2345 sec/batch
Epoch 10/10 Iteration: 43000 Avg. Training loss: 3.7530 0.2353 sec/batch
Nearest to time: the, in, paleontological, was, domination, sync, same, beyond,
Nearest to his: he, him, returned, father, career, son, degas, to,
Nearest to on: the, an, was, is, which, of, various, kamchatsky,
Nearest to many: used, most, in, are, particularly, of, known, some,
Nearest to was: the, had, by, as, but, were, of, on,
Nearest to that: to, the, is, it, and, in, a, for,
Nearest to states: united, first, of, faring, america, unrealized, ocogs, kacl,
Nearest to known: both, the, in, by, their, many, is, malum,
Nearest to active: volcanism, dissolves, facilitated, saavedra, oxidants, nami, often, sippar,
Nearest to existence: nauvoo, heist, eac, outcroppings, strontium, azrael, topkapi, bolland,
Nearest to hold: dowry, hypothesis, set, eminem, initialization, darin, specify, splintering,
Nearest to pressure: pools, volleyball, berbers, sublimation, impartiality, neuroanatomy, internal, fervently,
Nearest to bible: sevenfold, editions, masoretes, reaches, transcribed, therapeutae, redactor, ipf,
Nearest to joseph: ancient, puente, fenwick, colloquy, caspian, mendelssohn, selina, meir,
Nearest to gold: silver, oda, calamine, loft, viticulture, risings, leonis, refinery,
Nearest to except: they, shipped, humphries, apportioned, arboreal, sundry, apart, prokaryotic,
Epoch 10/10 Iteration: 43100 Avg. Training loss: 3.7444 0.2382 sec/batch
Epoch 10/10 Iteration: 43200 Avg. Training loss: 3.7850 0.2365 sec/batch
Epoch 10/10 Iteration: 43300 Avg. Training loss: 3.8140 0.2411 sec/batch
Epoch 10/10 Iteration: 43400 Avg. Training loss: 3.7639 0.2356 sec/batch
Epoch 10/10 Iteration: 43500 Avg. Training loss: 3.7677 0.2347 sec/batch
Epoch 10/10 Iteration: 43600 Avg. Training loss: 3.8627 0.2362 sec/batch
Epoch 10/10 Iteration: 43700 Avg. Training loss: 3.7863 0.2360 sec/batch
Epoch 10/10 Iteration: 43800 Avg. Training loss: 3.7938 0.2344 sec/batch
Epoch 10/10 Iteration: 43900 Avg. Training loss: 3.8163 0.2342 sec/batch
Epoch 10/10 Iteration: 44000 Avg. Training loss: 3.7873 0.2326 sec/batch
Nearest to time: in, the, was, after, sync, beyond, paleontological, terminates,
Nearest to his: he, him, returned, father, son, s, had, prodigy,
Nearest to on: the, was, is, which, an, and, a, various,
Nearest to many: used, most, are, particularly, known, in, candidiasis, as,
Nearest to was: had, the, as, by, on, which, in, and,
Nearest to that: to, the, it, is, for, and, in, these,
Nearest to states: united, first, america, of, faring, unrealized, u, kacl,
Nearest to known: both, in, the, was, by, their, as, malum,
Nearest to active: volcanism, dissolves, often, saavedra, facilitated, nami, facilitate, oxidants,
Nearest to existence: nauvoo, outcroppings, strontium, topkapi, eac, bitterness, creator, azrael,
Nearest to hold: the, hypothesis, dowry, eminem, set, initialization, and, darin,
Nearest to pressure: pools, volleyball, internal, impartiality, berbers, sublimation, fervently, jeff,
Nearest to bible: editions, sevenfold, reaches, exhorted, transcribed, masoretes, insights, mullis,
Nearest to joseph: puente, ancient, colloquy, fenwick, waitakere, mendelssohn, meir, hardly,
Nearest to gold: silver, oda, loft, calamine, leonis, risings, viticulture, waiver,
Nearest to except: they, shipped, humphries, arboreal, demobilized, apportioned, vr, prokaryotic,
Epoch 10/10 Iteration: 44100 Avg. Training loss: 3.7974 0.2362 sec/batch
Epoch 10/10 Iteration: 44200 Avg. Training loss: 3.8414 0.2335 sec/batch
Epoch 10/10 Iteration: 44300 Avg. Training loss: 3.8028 0.2359 sec/batch
Epoch 10/10 Iteration: 44400 Avg. Training loss: 3.7942 0.2376 sec/batch
Epoch 10/10 Iteration: 44500 Avg. Training loss: 3.8333 0.2369 sec/batch
Epoch 10/10 Iteration: 44600 Avg. Training loss: 3.8243 0.2369 sec/batch
Epoch 10/10 Iteration: 44700 Avg. Training loss: 3.8110 0.2353 sec/batch
Epoch 10/10 Iteration: 44800 Avg. Training loss: 3.8134 0.2358 sec/batch
Epoch 10/10 Iteration: 44900 Avg. Training loss: 3.7821 0.2341 sec/batch
Epoch 10/10 Iteration: 45000 Avg. Training loss: 3.7726 0.2347 sec/batch
Nearest to time: the, in, sync, was, beyond, eight, after, paleontological,
Nearest to his: he, him, returned, father, son, toulon, career, to,
Nearest to on: the, an, is, which, was, of, a, various,
Nearest to many: most, used, are, as, their, some, of, candidiasis,
Nearest to was: had, the, as, of, by, but, and, on,
Nearest to that: the, to, and, it, for, is, in, a,
Nearest to states: united, first, of, america, state, faring, unrealized, u,
Nearest to known: the, both, in, was, as, by, malum, their,
Nearest to active: volcanism, facilitated, often, saavedra, dissolves, oxidants, nami, facilitate,
Nearest to existence: nauvoo, topkapi, eac, outcroppings, azrael, strontium, heist, creator,
Nearest to hold: dowry, hypothesis, eminem, favor, casings, initialization, set, darin,
Nearest to pressure: pools, volleyball, internal, berbers, impartiality, fervently, sublimation, salute,
Nearest to bible: sevenfold, transcribed, reaches, editions, exhorted, fertile, masoretes, ipf,
Nearest to joseph: puente, ancient, mendelssohn, colloquy, fenwick, usk, thyself, waitakere,
Nearest to gold: silver, oda, loft, calamine, leonis, viticulture, risings, figs,
Nearest to except: they, shipped, humphries, demobilized, in, word, apportioned, sundry,
Epoch 10/10 Iteration: 45100 Avg. Training loss: 3.8216 0.2372 sec/batch
Epoch 10/10 Iteration: 45200 Avg. Training loss: 3.7892 0.2369 sec/batch
Epoch 10/10 Iteration: 45300 Avg. Training loss: 3.8963 0.2367 sec/batch
Epoch 10/10 Iteration: 45400 Avg. Training loss: 3.8837 0.2351 sec/batch
Epoch 10/10 Iteration: 45500 Avg. Training loss: 3.8381 0.2354 sec/batch
Epoch 10/10 Iteration: 45600 Avg. Training loss: 3.7690 0.2364 sec/batch
Epoch 10/10 Iteration: 45700 Avg. Training loss: 3.7423 0.2353 sec/batch
Epoch 10/10 Iteration: 45800 Avg. Training loss: 3.8058 0.2356 sec/batch
Epoch 10/10 Iteration: 45900 Avg. Training loss: 3.7149 0.2405 sec/batch
Epoch 10/10 Iteration: 46000 Avg. Training loss: 3.8501 0.2386 sec/batch
Nearest to time: in, the, sync, beyond, paleontological, was, domination, matem,
Nearest to his: he, him, returned, father, son, degas, to, had,
Nearest to on: the, was, of, is, and, a, an, with,
Nearest to many: used, more, as, most, are, some, and, of,
Nearest to was: had, the, as, on, of, but, by, first,
Nearest to that: the, it, to, and, is, in, for, a,
Nearest to states: united, first, of, america, faring, kacl, mee, ocogs,
Nearest to known: the, both, in, as, was, by, malum, is,
Nearest to active: volcanism, saavedra, dissolves, nami, facilitated, organization, member, oxidants,
Nearest to existence: nauvoo, eac, outcroppings, topkapi, pacifists, knapsack, azrael, strontium,
Nearest to hold: dowry, darin, favor, hypothesis, eminem, impacting, set, casings,
Nearest to pressure: pools, volleyball, berbers, internal, fervently, impartiality, salute, sublimation,
Nearest to bible: sevenfold, reaches, transcribed, editions, exhorted, masoretes, ipf, insights,
Nearest to joseph: puente, ancient, mendelssohn, colloquy, fenwick, thyself, bundling, colonies,
Nearest to gold: silver, oda, loft, calamine, leonis, viticulture, precious, medal,
Nearest to except: they, apart, humphries, shipped, sundry, bando, demobilized, apportioned,
Epoch 10/10 Iteration: 46100 Avg. Training loss: 3.8297 0.2399 sec/batch
Epoch 10/10 Iteration: 46200 Avg. Training loss: 3.7916 0.2369 sec/batch
Restore the trained network if you need to:
In [23]:
with train_graph.as_default():
saver = tf.train.Saver()
with tf.Session(graph=train_graph) as sess:
saver.restore(sess, tf.train.latest_checkpoint('checkpoints'))
embed_mat = sess.run(embedding)
Below we'll use T-SNE to visualize how our high-dimensional word vectors cluster together. T-SNE is used to project these vectors into two dimensions while preserving local stucture. Check out this post from Christopher Olah to learn more about T-SNE and other ways to visualize high-dimensional data.
In [24]:
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
import matplotlib.pyplot as plt
from sklearn.manifold import TSNE
In [25]:
viz_words = 500
tsne = TSNE()
embed_tsne = tsne.fit_transform(embed_mat[:viz_words, :])
In [26]:
fig, ax = plt.subplots(figsize=(14, 14))
for idx in range(viz_words):
plt.scatter(*embed_tsne[idx, :], color='steelblue')
plt.annotate(int_to_vocab[idx], (embed_tsne[idx, 0], embed_tsne[idx, 1]), alpha=0.7)
Content source: 3upperm2n/notes-deeplearning
Similar notebooks: